Handling Git Large repositories

Hey Hello dear DevOps and DevSecOps and SRE team heros!! here I came new challenge to solve the common problem on the Git. You may be using GitHub or GitLab or even BitBucket for source code management, Now few projects websites or mobile apps requires to store images, audio files, or video files which are larger in size. During the transfer to the client systems they are facing following issues:

1. Slowness in git clone and fetch operations
2. Sluggish commits and status checks
3. Repository size bloat
4. Complexity in managing multiple branches

Git LFS installation

apt install -y git-lfs

Git LFS (Large File Storage) is used to handle large files in a Git repository efficiently by replacing them with lightweight pointers while storing the actual files in a separate location. Here are various examples of using Git LFS, including tracking, untracking, checking status, and more: ---

1. Initialize Git LFS

Before using Git LFS in a repository, initialize it:

git lfs install

This sets up Git LFS for the repository. ---

2. Track Large Files

To track specific file types, use:

git lfs track "*.psd"

or track a specific file:

git lfs track "bigfile.zip"

This updates the `.gitattributes` file to include:

***.psd filter=lfs diff=lfs merge=lfs -text

After tracking, commit the `.gitattributes` file:

git add .gitattributes
git commit -m "Track large files with Git LFS"

---

3. Check Tracked Files

To see which files are being tracked by Git LFS:

git lfs track

---

4. Check LFS Status

To check which large files are modified or committed:

git lfs status

---

5. Untrack a File

If you no longer want a file to be tracked by Git LFS:

git lfs untrack "bigfile.zip"

This removes it from `.gitattributes`. Then commit the change:

git add .gitattributes
git commit -m "Untrack bigfile.zip from Git LFS"

Important: This does not remove files from previous commits. ---

6. List LFS Objects

To see which LFS files exist in your repository:

git lfs ls-files

---

7. Migrate Large Files (if added before tracking)

If you accidentally committed a large file before tracking it with Git LFS, migrate it:

git lfs migrate import --include="*.zip"

---

8. Push and Fetch LFS Files

After committing, push LFS files to the remote:

git push origin main

To pull and fetch LFS files:

git pull
git lfs fetch

---

9. Removing LFS Files From History (if needed)

If a large file was added before tracking and you want to remove it:

git lfs migrate import --include="bigfile.zip" --everything

Then, force push the cleaned history:

git push origin --force --all

---

10. Verify LFS Files in Remote Repository

To check which LFS files exist on the remote:

git lfs ls-files --long

Common mistakes working with Git LFS

• Not Tracking Files Properly: Forgetting to use git lfs track for large files can lead to them being stored in the repository instead of being managed by Git LFS.
• Ignoring the .gitattributes File: The .gitattributes file is crucial for Git LFS to function correctly. Failing to commit this file can cause issues for collaborators.
• Pushing Without Installing Git LFS: If Git LFS isn't installed on your system, pushing large files will fail or result in errors.
• Exceeding Storage Limits: Platforms like GitHub have storage limits for Git LFS. Exceeding these limits can block further uploads.
• Cloning Without Git LFS: If you clone a repository without Git LFS installed, you might end up with pointer files instead of the actual large files.
• Using Git LFS for Small Files: Git LFS is designed for large files. Using it for small files can unnecessarily complicate your workflow.
• Not Cleaning Up Old Files: Over time, unused large files can accumulate in the LFS storage, increasing costs or storage usage.

Please write your learnings in the comments.

Git installation on Ubuntu

Git installation on Ubuntu is pretty simple. If you are looking for git instatlling on the RHEL or Rocky or Oracle Linux you can use this link.

Now most of the software projects are using Git. So let's do installation on Ubuntu wit the following steps:

1. Check for Git exists
2. Install Git
3. Confirm Git Installation

Pre-requisites:

Pick an instance on Cloud or online terminal of Ubuntu 20+ version to this experiment. Here I'm using the KillerCoda provided Ubuntu.

Check for Git exists

This is a common requirement when you join a new project and on the Linux machine you would like to know git installed or not. We have couple of options to check it. Let's do it here:

dpkg -l git

#or

dpkg --list git

In the output first 'ii' in the list means (if there are packages installed, you should see this mark) that the package is correctly installed and available. alternatively you can also try other option to check git installation on Ubuntu.

apt list git

#or

apt list git -a

Or else you can simply use `git` it will works and provides git command help when git already installed.

git

We can see that it is not the latest version as of now, So I wish to get the latest git version on Ubuntu.

Install Git

1. we will download our desired verison of Git tarball, untar it, enter into the extracted directory. 2. Build the source code using make 3. Run the installation using make install

# install pre-requisite libraries 
sudo apt update
sudo apt install make libssl-dev libghc-zlib-dev libcurl4-gnutls-dev libexpat1-dev gettext unzip

# Download your desired version
wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.48.1.tar.gz
tar -zxf git-*.tar.gz 

cd git-2.48.1
make prefix=/usr/local all
sudo make prefix=/usr/local install

Now Build steps

Confirm Git Installation

You know already multiple ways ot check git existing on your Ubuntu. Let's quickly validate it with `--version` opton.

git --version

Auomation in mind

Power of your brain should work smart when you look at this instructions any AI tool can gives these but we have God's given our own brain to use :)

#!/bin/bash

SECONDS=0
GIT_VERSION="2.48.1"

if ! sudo apt update; then
    echo "Failed to update package list"
    exit 1
fi

rm -rf git*
sudo apt install -y make libssl-dev libghc-zlib-dev libcurl4-gnutls-dev libexpat1-dev gettext unzip

wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-$GIT_VERSION.tar.gz

tar -zxf git-*.tar.gz 
cd git-$GIT_VERSION
pwd

make prefix=/usr/local all
sudo make prefix=/usr/local install
git --version

  
git --version | grep ${GIT_VERSION}
[ $? -eq 0 ] && echo "Git installed latest version successfully!"
echo "Execution time: $SECONDS seconds"

Magical way of Git installation

The beatiful output of the above bash script is here:

Write your valuable comments with your experiments. Happy learning!!