Removing large files from git repository

Tags

git, repositories, service oriented architecture, shell script, svn

About a year ago, I setup git repsitoris for our projects and sub projects. Because I was leading the effort towards a Service Oriented Architecture, we decomposed the whole application in various services which could run independent. And being independent, they could be treated as separate projects and we setup separate git repositories.

Lately, I noticed that size of the repositories was pretty high around 300+ MB. Upon inspection, I discovered that large SQL dumps were being comitted and each change to them obviously took the toll in terms of space and the time it takes to clone/push and pull etc.

I wanted not only to remove the files but to totally remove them from the history as well. Here’s what I did:


$ git clone git@code.from.somewhere.com:repo.git # Get the repo
$ cd repo
$ # Remove the db/ and all files in it.
$ git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch .db'
$ rm -rf .git/refs/original/
$ git reflog expire --expire=now --all
$ git gc --prune=now
$ git gc --aggressive --prune=now
$ mv .git .. && rm -fr * # Now we make it a bare repository
$ mv ../.git .
$ mv .git/* .
$ rmdir .git
$ git config --bool core.bare true #
$ cd ..; mv repo repo.git # Done with making it bare, renaming just for clarity


Now copy the new repo.git in place of the existing on and that’s all.

Please note that git garbage collection not only might take too much time but also might consume enormous amounts of CPU if files were large enough.

Another issue might be that you imported from SVN and .svn directories are still there. In such case, find them all with the command below:


$find . -type d -name .svn


And apply the process for each of them. I know that’s tough, and I am looking to use xargs to do the stuff or turn it into a shell script.

CSS: The Floats

Tags

css web-design html xhtml

This post is about CSS floats. Well floats are pretty simple. There is nothing complicated about them. Complication only arises of two things. One is using them in a complex manner and the second is cross browser compatibility issues.

Let’s recap what we know about browsers. Every tag in HTML or XHTML basically creates an “element”. And that element can be either of two types when it comes to rendering. Either it can be an inline element or it can be block element.

Inline elements, when appear in succession, would be placed let to right (or right to left if page direction is right to left which can be set in the html tag’s dir attribute i.e dir=”rtl”) till the end of line.

While the block level elements sit on top of each other and expand horizontally to fully expand the space.

But we can do an interesting thing to a block level element. We can order it to float it self. But where it should float? We can ask it to float either on the right side or on the left side. So when we ask an element to float, it would be plucked out of the sequence in which it was previously appearing. And when plucked out then of course that space would be pretty much empty. What do we do with that space then? Well we do not have to do anything. If there is another element just after the one being floated, it would take the place of the floated element.

Astute reader may ask that because block level elements fully expand till the width available so if they are floated left or right, it would have no difference other then that the element after the floated one would be taking place of the floated element.

Quit correct. That is how it would be happening. But think of this. You have a block level element whose dimensions are 50px (i.e 50px in width and 50px in height). If order this element to float towards right (and we will see how to do that) then what will happen?

Well simply two things:

• The element’s right margin would be touching to the right boundary of the containing element
• The element after the floated element would be pulled 50px up because the space would be freed up by the floated element

So that is how floats work. There’s only one thing that is left about them. And that is that what if you do not want the element after the one being floated to come up and fill up the space? Yes that’s perfectly possible as well. We can do that and we will see in a while how.

Technically speaking

Suppose there is an element with class float-it and another element with class after-floated

So here’s the markup:

1. <p>A block level element. That is, paragraph</p>
2. <!-- This will be floated left
3. But we will apply little beautification to it:
4. * border would be set
5. * background-color would be set
6. * height and width would be set to 150 px
7. -->
8. <div class="float-it">50x50</div>
9. <!-- We will see the use of this later --->
10. <div class="after-floated">just floated</div>
11. <p>Another block level element</p>

And we apply the following CSS:

1. .float-it{
2. float:right;
3. }
6. /* This is just cosmetics */
9. body{
10. background-color: #CCC;
11. }
14. /*
15. * Border and background just for illustration
16. * While height and width just to demonstrate.
17. */
18. .float-it {background-color: #FCFC09;
19. width: 150px;
20. height: 150px;
21. border: 1px solid black;
22. }
23. .after-floated{border: 1px solid green;}

So as we have set the float property to right, the box is pushed to right. Possible values for float property are:

float: left|right|none

none is useful if an element is already floating and you want to turn that off.

Conclusion

So that was about floats. And its only that much believe me. Only one thing that I have not touched is clearing which is even more easier then floats. Let’s recap about the floats. We have a float property with which we can ask an element to float itself either to the left or right side of the containing block (that can be body, or the div in which that element appears). The element just after the floated element would be pulled up to take up the space because the element above it is now floating and no more in the document flow.

Ubuntu Karmic and Intel 82915G/GV/910GL Integrated Graphics Controller

Tags

graphics, intel, linux, resolution, ubuntu

I am using Ubuntu for almost two years both at home and office. My office setup quit complex in this regard as we have nVidia GeForce cards with dual LCD monitors on our disposal.  And at home, I am using my HP system which has Intel 82915G/GV/910GL Integrated Graphics Controller. I had some issues regarding resolutions but I almost totally forgot about them and upgrades after upgrades, all worked fine.

But things changed after the Ubuntu Karmic, I was stuck to only 2 resolutions: 640×480 and 800×600 both at 60Hz while my chipset and my monitor are both far more capable then this. In this post, I would try to list how I solved this problem for myself. I am no Linux geek and have a pure Windows background so you know who to blame if something seems odd.

In order get a working video setup, you need find out:

1. What video adaptor do you have?
2. Is there a driver for it?
3. Is it picking up the modes and resolutions if answer to above two questions is positive?
4. What your monitor is capable of?
5. Is the monitor’s capabilities are known to the system?

I assume that you have very little knowledge of the Linux stuff just like me. Lets try to get an answer for all these questions one by one.

What video adaptor do you have?

Well, open up command shell by pressing Alt + F2 and typing x-terminal-emulator
Probably you would be aware that there is a command available to get the listings of a directory. That command is called ls. In the same way, you can also get a list of the devices attached to PCI slots by a command lspci. If you issue this command, you would get an information about all the components on your system. But probably we are not interested in all the devices right now but only in our VGA. So therefore, issue this command:
lspci | grep VGA

and my output is:
00:02.0 VGA compatible controller: Intel Corporation 82915G/GV/910GL Integrated Graphics Controller (rev 04)
Little explanation about the command. lspci‘s output is given to another program called grep and we are telling grep to show only those lines where string VGA. This mechanism is called piping where output of one program becomes input of the other and thus you can form chain of process to do wonderful stunts. A UNIX philosophy.
At this step, you know what kind of VGA do you have. Next is to determine if you have driver for it or not.

Is there a driver for it?

If you have ATI or nVidia graphic adopters, then your output might be different. However, refer to this page about drivers. For my case, I did following:

aptitude search intel

And my output looked like this:

p intel-gpu-tools - tools for debugging the Intel graphics driver
p intel-microcode - Processor microcode data file for Intel CPUs
v intel-rng-tools -
p intel2gas - A converter from NASM assembly language to GAS
p inteltool - Dump Intel CPU / chipset configuration parameters
i A libdrm-intel1 - Userspace interface to intel-specific kernel DRM services -- runtime
p libdrm-intel1-dbg - Userspace interface to intel-specific kernel DRM services -- debugging symbols
i xserver-xorg-video-intel - X.Org X server -- Intel i8xx, i9xx display driver
p xserver-xorg-video-intel-dbg - X.Org X server -- Intel i8xx, i9xx display driver (debug symbols)


A little explanation of what I did and why. Well I need to know if drivers for my card are installed or not. They would have intel somewhere in there name. aptitude is a program that controls all the software installations on your system and automatically keeps them up to date. So there is a large repository of software behind it somewhere on internet (well Ubuntu servers). This concept of single source of software is unknown to commercial world because they have “copy rights” not in there hand and that’s why it is an alien concept to the user of windows. Of course you cannot hope to install CorelDraw this way!. Well anyway, what I am doing is that I search for a package that name contains intel and I get a list of all those packages. Look at this line:

i xserver-xorg-video-intel - X.Org X server -- Intel i8xx, i9xx display driver

The i in front of that line indicates that this package is already installed. If it were not, instead of i, I would have a p there. Suppose if it w to the deviceere not, I would install it like this:

sudo aptitude install xserver-xorg-video-intel

Is it picking up the resolutions?

If you’ve got all well as of now, that’s great. Now we need to determine that if the drivers are really working with the VGA. We do this with two utilities. xrandr and cvt

Now we need to experiment with the video adapter. How? We need to check if it supports those resolution we expect from it. For example, I expect that my chip should show up to 1024 760 at 85 Hz. How to check this then? Its in three steps:

1. Introducing a new mode
2. Associating that mode to a device
3. Applying that mode on that device

This process is better explained here

<h2>What your monitor is capable of?</h2>

Well this might be most important question. You need to know the maximum resolutions, optimal resolution and horizontal and vertical frequencies if your monitor is not automatically detected as it was with me. I happen to beusing Sony 21″ Multiscan E500 CRT monitor which is capable of displaying upto 2048×1536 at 60 Hz while recommended resolution is 1600×1200 at 85 Hz. The Horizontal frequency range is 30-109kHz and 48-160Hz for vertical. Note that horizontal frequency is in kHz and vertical is in Hz. If you are curious then you can look about CRT workings here. Anyway, Used this information in my xorg.conf file which is at /etc/X11/xorg.conf and added following:


Section "Monitor"
Identifier "Sony Multiscan E500 CRT Monitor"
VendorName "Sony"
ModelName "CPD-E500"
HorizSync 30-109
VertRefresh 48-160
Option "DPMS"
EndSection

Section "Screen"
Identifier "Default Screen"
Monitor "Sony Multiscan E500 CRT Monitor"
Device "Configured Video Device"
DefaultDepth 24
SubSection "Display"
Modes "1600x1200"
Modes "1024x768"
Modes "800x600"
EndSubSection
EndSection


while my complete xorg.conf looks like this:

# xorg.conf (X.Org X Window System server configuration file)
#
# This file was generated by dexconf, the Debian X Configuration tool, using
# values from the debconf database.
#
# Edit this file with caution, and see the xorg.conf manual page.
# (Type "man xorg.conf" at the shell prompt.)
#
# This file is automatically updated on xserver-xorg package upgrades *only*
# if it has not been modified since the last upgrade of the xserver-xorg
# package.
#
# If you have edited this file but would like it to be automatically updated
# again, run the following command:
# sudo dpkg-reconfigure -phigh xserver-xorg

# commented out by update-manager, HAL is now used
#Section "InputDevice"
# Identifier "Generic Keyboard"
# Driver "kbd"
# Option "XkbRules" "xorg"
# Option "XkbModel" "pc105"
# Option "XkbLayout" "us"
#EndSection

# commented out by update-manager, HAL is now used
#Section "InputDevice"
# Identifier "Configured Mouse"
# Driver "mouse"
# Option "CorePointer"
#EndSection

Section "Device"
Identifier "Configured Video Device"
Option "UseFBDev" "true"
EndSection

Section "Monitor"
Identifier "Sony Multiscan E500 CRT Monitor"
VendorName "Sony"
ModelName "CPD-E500"
HorizSync 30-109
VertRefresh 48-160
Option "DPMS"
EndSection

Section "Screen"
Identifier "Default Screen"
Monitor "Sony Multiscan E500 CRT Monitor"
Device "Configured Video Device"
DefaultDepth 24
SubSection "Display"
Modes "1600x1200"
Modes "1024x768"
Modes "800x600"
EndSubSection
EndSection

Section "ServerLayout"
Identifier "Default Layout"
Screen "Default Screen"
EndSection


After this, restart your display manager in my case it was gdm so I did:
sudo /etc/init.d/gdm restart

And from that point onwards, I had no issues with video resolution. Next on my target is sound. Thought it is working but I would document some of my discoveries about it.

Why we’ve not grown?

Tags

business, society

The problem what I’ve been thinking about. Why we’ve not out grown to a visible stage? Why dont we have large and huge business empires? Why is it so that most of our bussinessmen are confined to the same country and not beyond?

Honesty

Yes that’s the factor. Even in our private business, our managers would favour those with who are their favourates and not the talented people around. One of my friend in a large Telecom and Internet provider company, told me of such things as of how low calliber people get raised up just because they’e got links.

In a software company, I’ve even heard of that and as result all the talented started to move away and out. Definately that affects the business then.

You might be interested in knowing why one prefers dull people over shine? Mostly fear. Fear of being superseded that would force them to make “safe and smart” choices. But think of it what gets sorted out as result of this “bubble” sort?

So am I asserting that all the managers and those responsible in organizations are dishonest? Then how the hell these businesses are going after all? No, they are not dishonest. They use “selective honesty” which in turn haunts explosive growth as you’ve not let the best to be the next.

So this algorithm being executed in all the organizations of the whole state yields dull bodies and overall we’re a failed state.

Think about it. If you’ve got your own business, or intend to initiate one, beware of the managers who are “slightly” dishonest and even do not do it yourself if you’re at decision making. If managed to do so, then that’s for sure that you’re going to flourish to the maximum. No doubts about it.

The Questions I have been wondering about

From time to time, I’ve been wondering about many questions in different phases of life. Here are some:

• What is power? How it comes? Why they bow to someone and not to the other?
• How human race continued to grow in different domains even being unaware of the other areas being developed and dug by fellow human? Where do they overlap?
• Introverts are also inventors? How?
• How large business empires work? What keeps them tied and focus on the goal? What the top level is getting?

I’d keep it updating as I come across more.

The Anne Frank

I just got a book on my cell phone and kept reading as long as I could: The Diary of a young girl. Yet not in the middle even, I got curious to know if the diary is for real? Reasons for the doubt were the style of writing and the observation that girls makes about herself and the others not seem to be from a 13 years old girl.

I’ll update about it as I got along.

A mimimal JavaScript Logger

Tags

closures, javascript, JSLint, logger, UrduTools

For one of my internal projects, I do not wanted to rely any of the JavaScript frameworks out there and do all on my own. On Linux with Firefox and Firebug installed, I was writing some information to firebug console. When the application ran on another browser without Firebug, it simply did not work at all.I made it working by commenting out the logging calls for the moment but that was not the solution.

My first idea was to call window.console if it is not undefined. But that would not be something nice and would be a lot of code. The scene was clear. I had to write my own logger but what should it do? I thought it must check if window.console is available and emit message there and if not, should silently ignore.

I started to work on the idea and while working, I thought why not try to emit messages to an HTML div in case window.console is not available? So I worked on that. In first iteration you had to instantiate logger yourself. Then I made it already instantiated and you’d had to call the methods only. The whole idea heavily relies on JavaScritpt anonymous functions, inner functions and JavaScript closures.

Here’s the source code:

/*
 *The urdu tools base file
 */
if(typeof(URDUTOOLS) === 'undefined' || !URDUTOOLS){
    URDUTOOLS = {};
}

if(typeof(URDUTOOLS.util) === 'undefined' || !URDUTOOLS.util){
    URDUTOOLS.util = {};
}

if(typeof(URDUTOOLS.util.Logger) === 'undefined' || !URDUTOOLS.util.Logger){
    /**
     * Our own logging facility. It basically searches for a special DIV with
     * ID URDUTOOSL_logger_dock. If such a DIV is found, its used as logging 
     * area. If that is not found, window.console is used (which is available in
     * case of Fiebug installed) and even if that's not available, you can set
     * one of your own like this:
     * 
     * URDProblem at line 78 character 10: Missing semicolon.UTOOLS.util.Logger.setLoggingDock("my_logging_div_id");
     * 
     * Logger.info()
     * Logger.warn()
     * Logger.debug()
     * Logger.error()
     * Logger.clear()
     * Logger.setLoggingDock(element_id)
     * Logger.getLoggingDock()
     *
     * Heavily dependent on closures.
     *
     */
    
    // Create a new Logger object at the spot!
    URDUTOOLS.util.Logger = new (function(div_id){
    
        // Yet this is null
        var logging_dock = null;
        
        // Logger.info()
        this.info = function(msg){
            logging_dock = this.getLoggingDock();
            if(logging_dock){
                logging_dock.info(msg);
                }
        };
        
        // Logger.warn()
        this.warn = function(msg){
            logging_dock = this.getLoggingDock();
            if(logging_dock){
                logging_dock.warn(msg);
                }
        };
        
        // Logger.debug()
        this.debug = function(msg){
            logging_dock = this.getLoggingDock();
            if(logging_dock){
                logging_dock.debug(msg);
                }
        };
        
        // Logger.error()
        this.error = function(msg){
            logging_dock = this.getLoggingDock();
            if(logging_dock){
                logging_dock.error(msg);
                }
        };
        
        // Logger.clear()
        this.clear = function(){
            logging_dock = this.getLoggingDock();
            if(logging_dock){
                logging_dock.clear();
                }
        };
        
        // Logger.setLoggingDock()
        // We try to get the given DOM element and extent it with some 
        // of our own methods.
        this.setLoggingDock = function(el_id){
            var element = document.getElementById(el_id);
            
                // Inner function to build log message
            function logMsg(type, msg){
                return "<b>"+ type + "</b>: " + msg + "<br/>";
             }
                
            if(element){
                logging_dock = element;
                }
            
            if(logging_dock){
            
                logging_dock.info = function(msg){
                    this.innerHTML += logMsg("INFO", msg);
                };
            
                logging_dock.warn = function(msg){
                    this.innerHTML += logMsg("WARN", msg);
                };
            
                logging_dock.debug = function(msg){
                    this.innerHTML += logMsg("DEBG", msg);
                };
            
                logging_dock.error = function(msg){
                    this.innerHTML += logMsg("EROR", msg);
                };
                
                logging_dock.clear = function(){
                  this.innerHTML = "";  
                };
            }
        };
        
        // Logger.getLoggingDock()
        
        this.getLoggingDock = function(){
            // Get current div_id DIV if logging_dock not set
            // This is basically becasue when the new() logger is being 
            // execute for the first time, the DIV to be used for logger is not
            // available as yet. Each of the logging methos first always get
            // the logging dock by calling this method. If the DOM is ready,
            // and element is get once, its not get for the next times.
            // Read the code carefully to understand!
            if(!logging_dock){
              this.setLoggingDock(div_id);
              }
            if(!logging_dock){
              logging_dock = window.console;
              }
          return logging_dock;
        };
        
    })("URDUTOOLS_logger_dock");
}

After writing this, I ran it through JSLint and found no problems and it works well for me.

The Choice of JavaScript

Tags

appsengine, javascript, python, ruby

For one of my internal private pet research projects pertaining to natural language processing, I ended up in implementing it with JavaScript. The factors that contributed to this was the fact that I needed someone non-programmer to review the process and validate. My choices were being platform independence. I wished to built it as a Ruby on Rails application but setting up and running a Rails application for a naive user is not an easy task.

The second choice was Java and building it as a desktop applicaiton. That too was ruled out as running a Java application requires JRE to be installed and the potentail users might not be even capable of that.

I knew it to be a web application. I came accross Google Apps Engine. So the choice turned out to be Python. But looking at that platform, I thought that an appliation with a long future, it would not be wise to setup there as its future is yet not known. Moreover, its very restrictive in its data model. It would be pain to port it somewhere else later. I wanted to use CherryPy or Django for the web framework but again dropped the idea.

PHP is again not what I wanted. I had written few very small PHP scripts back in 2004 or earlier but never returned to it reason being that I wanted to be a Desktop application developer so keep investing myself into C/C++, Win32, MFC and .NET

But eventually, as I was learning advanced JavaScript, and I was also delivering a course in my company, I simply started to write it in JavaScript as it gives true platform independence with 0 installation overhead! Other reason was that I came accross some Japaneese projects that implemented whole Java and Ruby Virtual Machines in pure JavaScript. So I settled with implementing it in JavaScript.

I’m though enjoying writing it, but one of my concerns is obfuscation. For sometime, unless I am fully done with my idea, I do not want it to be stolen or worked out elsewhere and yet I am not able to find any free and perfect tool. May be I have to write that too. Let’s see. Let me know if you know any good one for compacting multiple JS files into one with obfuscation in place.

Books that I love

Tags

Books

There are literally thousands of books on computer science But fewer are that have hit me so deep that I refer to them as near to Bible on those subjects.

• Turbo C++ by Robert Lafore
• Object Oriented JavaScript by Stoyan Stefanov
• Object Oriented C (its a free book)

There are many more on my wish list that I want to read but dont have time for like Code Complete, Patterns of Enterprise Architecture. I’ll keep sharing my thoughts on this.

REST API: My experience and intents

I’m yet continuously thinking about my career. How do I grow it? And I’ve got many many things to do which I must undertake one at a time.

The current happening in my life is that of REST. I’ve been building a REST API for Dedomenon. Dedomenon is an online structured data storage engine. I was asked to build REST API on top of it. Though I’ve done the job very well as I far as I think but still I’d say that designing and API can never be complete until you put it to real use.

When it comes to real usage, You’re more likely to discover further hidden and indirect use cases which with the designer glasses, you wont be able to see. The concept of REST itself is not very old as it came just in 2000. Incorporation of this as the basis for providing web services in Ruby on Rails only started until 2006. Later, it became clear that Rails was going to favor REST over all the other floating paradigms.

This actually leads to a situation where there’s very little documentation on REST in Rails. Though there’s much but its pretty basic. Building a whole REST API is really something different. When I started out with this for Dedomenon, I felt there must be a short booklet about building REST APIs on top of Rails. An outline that comes to my mind is as under:

• What is REST?
• How do I Identify resources?
• How do I Identity operations on resources?
• How do I map my operations to HTTP verbs?
• Some tools of trade
• How do I expose resources? (Routes etc)
• Who’ll handle those requests? (Possible suggestions, engines plugin, etc)
• How do I response in multiple formats
• Doing so in multiple formats
• Accepting and processing multiple formats
• Writing tests for your REST API
• How do I implement REST Auth? (possible schemes)
• Managing access rights etc
• Advanced topics: Managing consumption rate and other issues

I’m thinking to write all about this in less then 100 pages. But I’m feeling that 100 pages would not be perhaps a doable amount for me. When you sit down to write, only then you realize that you’re not able to write even 25 pages!
Another decision is that I wan to use vim as text editor. For publishing it, either LaTeX or DocBook. Let’s see what do I choose. Most probably LaTeX would be the choice. What do you say?