Handling Git Large repositories

Hey, hello, dear DevOps, DevSecOps, and SRE team heroes!! Here I came across a new challenge to solve the common problem on Git. You may be using GitHub or GitLab or even Bitbucket for source code management. Now, a few projects, websites, or mobile apps require storing images, audio files, or video files that are larger in size. During the transfer to the client systems, they are facing the following issues:

1. Slowness in git clone and fetch operations: files taking too long to upload or download, leading to delays in deployment and user experience
2. Sluggish commits and status checks: some clients are encountering errors related to file size limitations, causing frustration and hindering workflow efficiency
3. Repository size bloat
4. Complexity in managing multiple branches

It's crucial for us to explore solutions that can streamline this process and ensure smooth handling of large media files.

Git LFS installation

apt install -y git-lfs

Git LFS (Large File Storage) is used to handle large files in a Git repository efficiently by replacing them with lightweight pointers while storing the actual files in a separate location. Here are various examples of using Git LFS, including tracking, untracking, checking status, and more: ---

1. Initialize Git LFS

Before using Git LFS in a repository, initialize it:

git lfs install

This sets up Git LFS for the repository. ---

2. Track Large Files

To track specific file types, use:

git lfs track "*.psd"

or track a specific file:

git lfs track "bigfile.zip"

This updates the `.gitattributes` file to include:

***.psd filter=lfs diff=lfs merge=lfs -text

After tracking, commit the `.gitattributes` file:

git add .gitattributes
git commit -m "Track large files with Git LFS"

---

3. Check Tracked Files

To see which files are being tracked by Git LFS:

git lfs track

---

4. Check LFS Status

To check which large files are modified or committed:

git lfs status

---

5. Untrack a File

If you no longer want a file to be tracked by Git LFS:

git lfs untrack "bigfile.zip"

This removes it from `.gitattributes`. Then commit the change:

git add .gitattributes
git commit -m "Untrack bigfile.zip from Git LFS"

Important: This does not remove files from previous commits. ---

6. List LFS Objects

To see which LFS files exist in your repository:

git lfs ls-files

---

7. Migrate Large Files (if added before tracking)

If you accidentally committed a large file before tracking it with Git LFS, migrate it:

git lfs migrate import --include="*.zip"

---

8. Push and Fetch LFS Files

After committing, push LFS files to the remote:

git push origin main

To pull and fetch LFS files:

git pull
git lfs fetch

---

9. Removing LFS Files From History (if needed)

If a large file was added before tracking and you want to remove it:

git lfs migrate import --include="bigfile.zip" --everything

Then, force push the cleaned history:

git push origin --force --all

---

10. Verify LFS Files in Remote Repository

To check which LFS files exist on the remote:

git lfs ls-files --long

Common mistakes working with Git LFS

• Not Tracking Files Properly: Forgetting to use git lfs track for large files can lead to them being stored in the repository instead of being managed by Git LFS.
• Ignoring the .gitattributes File: The .gitattributes file is crucial for Git LFS to function correctly. Failing to commit this file can cause issues for collaborators.
• Pushing Without Installing Git LFS: If Git LFS isn't installed on your system, pushing large files will fail or result in errors.
• Exceeding Storage Limits: Platforms like GitHub have storage limits for Git LFS. Exceeding these limits can block further uploads.
• Cloning Without Git LFS: If you clone a repository without Git LFS installed, you might end up with pointer files instead of the actual large files.
• Using Git LFS for Small Files: Git LFS is designed for large files. Using it for small files can unnecessarily complicate your workflow.
• Not Cleaning Up Old Files: Over time, unused large files can accumulate in the LFS storage, increasing costs or storage usage.

Please let me know your experiances with Large files on Git server, write your learnings in the comments.

GIT Commands reference for DevOps Engineer

Dear DevOps Engineer!! this post is full of experimenting with git commands running most of them on the 'Git Bash' window in my laptop. I've collected three things about each git command.

• git command syntax
• What this git command will do - short descriptions
• An example command that I've executed
• Screenshot of that command execution

How do I Start a Fresh repository in Git CLI(local repo)?

You need to create a folder where you would like to start your coding project. The code can be web-site related where you can have multiple directories involved such as HTML files into a folder, all images to be used in the website in a folder, CSS files into a folder, etc. First, make the folder structure for you project then at the root of the project need to initialize the project.  To create empty repo or re-initializing a repo

Syntax :

git init

Note: It is always better to have a fresh directory created and then run the above command to initialize a local Git repository.

git init example

Getting a remote repository to your git client - Local system/Laptop

You can have the remote repository either public or private teams on the following SCM clouds:

1. GitHub 
2. GitLab
3. BitBucket

git clone Syntax:

git clone ssh://git@github.com/[username]/[repository-name].git

Create a local copy of a remote repository of GitHub or Gitlab. Make sure that you're doing first time in that folder, it should not have the same repo that means if your re-run the same command it cannot work.

Please make sure you have the correct access rights and the remote repository exists

Example of using gitlab URL:

 git clone https://github.com/BhavaniShekhar/vtdevops2021feb.git

Vybhava : git clone 

Syntax:

git commit -m "[commit message]"

commit changes
Example:

git commit -m "some message"

git commit command

Syntax:

git checkout -b [branch name origin/[branch name]

Clone a remote branch and switch to it
Example:

git checkout -b feature2

Differences between git checkout & switch command

Syntax:

git checkout [branch name]

Switch to a branch
Example:

git checkout

git checkout command

Syntax :

git checkout -

Switch to the branch last checked out
Example :

git checkout -

git last checkout command

Syntax:

git checkout -- [file-name.txt]

Discard changes to a file on which you are worked recently.
Example:

Syntax:

git push origin [branch name]

push to a branch to your remote repository
 Example:

git push origin slave

git push command

Syntax:

git push -u origin [branch name]

push changes to remote repository (and remember the branch)
Example:

git push -u origin feature1

git push origin command

Syntax:

git push

push changes to remote repository (remembered branch)
Example:

git push

git push command 

Syntax:

git push origin --delete [branch name]

delete a remote branch
 Example:

git push origin --delete feature3

git remote branch delete oush command

git pull command

Syntax:

git pull origin [branch name]

update local repository to the newest commit
Example:

git pull

git pull command

pull changes from remote repository Example:

git pull origin newbranch2

git remote

Syntax:

git remote add origin ssh://git@github.com/[username]/[repository-name].git
    or
 git remote set-url origin ssh://git@github.com/[username]/[repository-name].git

add a remote repository Example 1:

git remote add origin git@gitlab.com:gitops4/docker-exercises.git

set a repository's origin branch to ssh Example 2:

git remote set-url origin git@gitlab.com:gitops4/docker-exercises.git

git log

The syntax for the git log command is:

git log [--summary] [--oneline]

view changes
Example:

git log

git log command

view changes made to files or directory in (detailed)
Example:

    git log --summary

git log summary command

view changes(briefly)
Example:

    git log --oneline

git log oneline command

Syntax:

git diff [source branch] [target branch]

preview changes before merging 

References: 

 Git log command examples https://www.thegeekstuff.com/2014/04/git-log/