Handling Git Large repositories

Hey, hello, dear DevOps, DevSecOps, and SRE team heroes!! Here I came across a new challenge to solve the common problem on Git. You may be using GitHub or GitLab or even Bitbucket for source code management. Now, a few projects, websites, or mobile apps require storing images, audio files, or video files that are larger in size. During the transfer to the client systems, they are facing the following issues:

1. Slowness in git clone and fetch operations: files taking too long to upload or download, leading to delays in deployment and user experience
2. Sluggish commits and status checks: some clients are encountering errors related to file size limitations, causing frustration and hindering workflow efficiency
3. Repository size bloat
4. Complexity in managing multiple branches

It's crucial for us to explore solutions that can streamline this process and ensure smooth handling of large media files.

Git LFS installation

apt install -y git-lfs

Git LFS (Large File Storage) is used to handle large files in a Git repository efficiently by replacing them with lightweight pointers while storing the actual files in a separate location. Here are various examples of using Git LFS, including tracking, untracking, checking status, and more: ---

1. Initialize Git LFS

Before using Git LFS in a repository, initialize it:

git lfs install

This sets up Git LFS for the repository. ---

2. Track Large Files

To track specific file types, use:

git lfs track "*.psd"

or track a specific file:

git lfs track "bigfile.zip"

This updates the `.gitattributes` file to include:

***.psd filter=lfs diff=lfs merge=lfs -text

After tracking, commit the `.gitattributes` file:

git add .gitattributes
git commit -m "Track large files with Git LFS"

---

3. Check Tracked Files

To see which files are being tracked by Git LFS:

git lfs track

---

4. Check LFS Status

To check which large files are modified or committed:

git lfs status

---

5. Untrack a File

If you no longer want a file to be tracked by Git LFS:

git lfs untrack "bigfile.zip"

This removes it from `.gitattributes`. Then commit the change:

git add .gitattributes
git commit -m "Untrack bigfile.zip from Git LFS"

Important: This does not remove files from previous commits. ---

6. List LFS Objects

To see which LFS files exist in your repository:

git lfs ls-files

---

7. Migrate Large Files (if added before tracking)

If you accidentally committed a large file before tracking it with Git LFS, migrate it:

git lfs migrate import --include="*.zip"

---

8. Push and Fetch LFS Files

After committing, push LFS files to the remote:

git push origin main

To pull and fetch LFS files:

git pull
git lfs fetch

---

9. Removing LFS Files From History (if needed)

If a large file was added before tracking and you want to remove it:

git lfs migrate import --include="bigfile.zip" --everything

Then, force push the cleaned history:

git push origin --force --all

---

10. Verify LFS Files in Remote Repository

To check which LFS files exist on the remote:

git lfs ls-files --long

Common mistakes working with Git LFS

• Not Tracking Files Properly: Forgetting to use git lfs track for large files can lead to them being stored in the repository instead of being managed by Git LFS.
• Ignoring the .gitattributes File: The .gitattributes file is crucial for Git LFS to function correctly. Failing to commit this file can cause issues for collaborators.
• Pushing Without Installing Git LFS: If Git LFS isn't installed on your system, pushing large files will fail or result in errors.
• Exceeding Storage Limits: Platforms like GitHub have storage limits for Git LFS. Exceeding these limits can block further uploads.
• Cloning Without Git LFS: If you clone a repository without Git LFS installed, you might end up with pointer files instead of the actual large files.
• Using Git LFS for Small Files: Git LFS is designed for large files. Using it for small files can unnecessarily complicate your workflow.
• Not Cleaning Up Old Files: Over time, unused large files can accumulate in the LFS storage, increasing costs or storage usage.

Please let me know your experiances with Large files on Git server, write your learnings in the comments.

GitHub Personal Access Token (PAT) for Linux Users

Hey Greetings of the day!!

GitHub providing Personal Access Token instead of using username, password for git repository that is on the GitHub. where git subcommands such as git pull, git push, fetch and any remote operations will be depends on this PAT. 

 There are two choices for PAT 

1. fine grain Personal Access Token(PAT) 

2. Personal Access Token (Classic) 

I understood that it is easy to change permissions, authentication token that use only specific target repositories. 

If you want to know 'How PAT works? Where to start PAT' then this post is absolutely for you! Welcome to this 2 mins read!

• Fine grained PAT newly introduced in Oct 2022 on the GitHub still at the time this post it is mentioned as [Beta] versioned.
• Personal Access Token PATS are going to work with commonly defined API on GitHub. Any integration made simplified with this method.

How to create PAT on GitHub? 

1. Login to your GitHub account. Click on the profile picture upper-right corner, from there select the 'Settings' menu item. 
2. The left bottom, PAT will be defined under the 'Developer Settings' there you will get new page with 'Personal Access Tokens' into the work area select your choice. There are many actions related to each row tells about a granular choices that GitHub could provide. 
3. Go to right side 'Generate Token' will be a button to hit. Then it will prompt you for login credentials to be entered. Then New personal access token parameters will need to be filled up. 1. Note - alias a authentication methods let's enter the note as "vt_demo_token" 
4. Select the scope - 'repo' is enough, where it will be containing repo:status, repo_deployment, public_repo, repo:invite, security_events.
5. Click on the "Generate Token" button new page will have the token. Make sure to copy your new personal access token now. Remember that you won't be able to see it again eventhough you have created it. As soon as click away of this page you will not get the value any more.

Preserving the TOKEN on your Linux system Create an environment variable store it inside .bash_profile.

MY_GITHUB_TOKEN=GENERATED-TOKEN-VALUE-COPIED FROM GitHub PAT page.

How do I access remote GitHub repository?
Use the following command to set the remote repository:

git remote origin https://github-remote-repo-url 

Use the following command to check the remote repository

git remote -v 

Here -v option indicates that verbose of the remote url and fetch and pull commands could connect with the URLS shown as above command output. push the changes to remote GitHub repository.

git push -u origin https://git-url 

It will prompt for the Password then enter the TOKEN value which is already copied that you can paste here. To store the password you don't want to see the prompting every push, pull activities. To avoice that you can set the remote url as follows:

git remote set-url https://api:$MY_GITHUB_TOKEN@github.com/REMOTE-USER/REMOTE-REPO.git 

Example:

git remote set-url https://api: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@github.com/SanjayDevOps2022/football.git

After setting this all remote operations push, pull are no user access prompted.
If you find this post is useful please share this to your technical communities :)

Git branching - local and remote

Hello all, in this post specially dedicated to all DevOps/DevSecOps Engineers.  Where their daily routine will have this commands if we know these we can play with them easily if you don't then it will be horrifies you! 

Git Branching explained in detailed

Prerequisites to this you should be aware git basics and to experiment with these command examples should have git installed on your Mac/Linux or git bash on Windows system.

What is the Branch in Git?

A Git branch is a separate line of work, where your work will not disturb other DevOps team members work.

Branch is a simply a movable  pointer to commit, this we can observe using git branch command output wherever you see that '*' prefixed there the pointer pointing. Default branch is master or main (new trend) but it is optional we can name anything when we initiate the git repository.

Main branch is the FIRST branch of your project that means once you run the 'git init' then 'git branch' you can observe this.

On the branch we can have many changes until find the solution and accordingly you need to do multiple commits.

Each branch created for different purpose for example new feature development we name them as 'feature'. similarly we can also have planned releases where we name them as 'release1.0' or 'dev-release'. Again it is flexible to the SCM manager decision.

Why we need to create branch in Git?

The Major reasons for Git branching are:

• Any new features development could BREAK the code, to save this.
• No disturbance between two developer codes that means this git branching simplify the collaboration work which gives more agility to the project

Gitting Start Branching

Here I'm with an interesting story about SCM Git branching that helps to work in collaborative environments where DevOps is a culture.

Let me share with you guys about the steps that I've executed in this are realtime requirements.

Top 10 Examples on git branch sub-command 

1. How to create a branch and check using git sub-commands?

To create a new branch from the existing branch. Here you should be on one of the branch. Generally we start from main or master branch.

git branch newbranch
git branch

2. How to jump onto the new branch directly when it is created?

Create a branch and switch to it in single command

git checkout -b feature
git branch

3. How to switch between the git branches?

Switch to branch We have two command options for switching between branches. 

a. Regular subcommand for switching to a branch

git checkout newbranch
git branch

b. New way to switch between the branch

git switch master
git branch

5. How can I view all local repo branches? 

To view all branches which are created in the local repository no option will do or else we can use status with -b to see in single-line output

git branch 
git status -b -s #alternative option

6. How can I see remote branches on my local laptop/local repo?

To view all the remote branches only you need to use -r or --remote as options

This is command will help you to get more info about all the branches are available in the remote git server such as GitHub or BitBucket etc.
 

git branch -r 
git branch --remote 

git branch remote to view your github or bitbucket branches

7. How can I see all local and remote repositories together?

To view all branches including remote branches use -a or --all options 

git branch -a 
git branch --all 

Observe the different colors for each repos.

1. The green color indicates current working tree(branch)
2. The red indicates remote repo branches
3. The white color indicate local branches

git branch all command example

8. Is there any way to delete a git branch in local repo? 

Delete a branch can be allowed when we use the -d or --delete options on a target branch.

Note : You cannot delete the branch on which you are working. Switch to anyother branch then run this.

git branch -d newbranch
git branch --delete newbranch

Removing a branch

[This is best option to delete because it preserves the recent changes made on the local branch] 9. Deleting the remote branch This is delicated operation, double check before you run this command.

git branch -D myremote-branch 

Compare -d and -D example for git branch removal

9. How to delete a remote branch in git command line ?

Better option is to use delete option with push command

git push origin --delete myremote-branch

10. How to rename a branch on git?

Move a branch/Rename 

a. Linux move command used to rename same way here a branch can be renamed and its reflog

git branch -m newbranch dev
git branch --move newbranch dev 

Git branch move or renaming

b. Rename forcefully 

Even though your target branch exist the git move will happen forcefully when we use -M option.

git branch -M newbranch dev

Hope you enjoy this story. Connect with our earlier technical articles might give some new tricks or tips for your project growth and more productive.

Cheers!!