Large repos and troubleshooting · Prasanta Kumar Behera

Keep big repositories fast with shallow, partial and sparse clones and Git LFS; choose between submodules and subtrees; and fix the errors engineers hit most, from rejected pushes to detached HEAD, line-ending noise and SSH authentication failures.

Keeping big repositories fast

Technique	What it saves	Use it for
Shallow clone `--depth 1`	History	CI jobs that build one commit
Partial clone `--filter=blob:none`	File contents of old commits (fetched on demand)	Developers in big repos who still want history
Sparse checkout	Files outside the folders you need	Monorepos: work only in `apps/payments`
Git LFS	Large binaries in history	ISOs, images, datasets that must be versioned
An artifact store instead	Binaries entirely	Build outputs: registries, S3, Artifactory

$ git clone --filter=blob:none --sparse git@github.com:acme/platform.git
$ cd platform
$ git sparse-checkout set apps/payments charts/payments

In CI, note that a shallow clone has no tags or older commits: tools that compute versions (git describe) need fetch-depth: 0 or an explicit git fetch --tags.

You don't carry the whole library home to read one chapter. A shallow clone borrows today's newspaper only; a sparse checkout takes just the shelves you need; LFS leaves the heavy atlases in the store room and gives you a ticket to fetch them when you really need one.

Submodules or subtrees?

Sometimes one repository needs another inside it (shared Terraform modules, a vendored chart):

	Submodule	Subtree
How	A pointer to a specific commit of another repo	The other repo's files copied in, with history merged
Pros	Exact version pinned, small	Just files: cloning and CI need nothing special
Cons	Extra commands (`--recursive`), easy to forget updates, confusing for newcomers	Updating and contributing back is more manual
Usually better	Pin versions via a package manager or registry instead (Terraform module registry, Helm repo, Go modules)

For infrastructure code, referencing modules by version from a registry or a Git tag (?ref=v3.4.0) is almost always cleaner than either.

The errors everyone hits

Error	Meaning	Fix
`! [rejected] main -> main (non-fast-forward)`	The remote has commits you don't	`git pull --rebase`, resolve, push
`You are in 'detached HEAD' state`	Checked out a commit/tag, not a branch	`git switch -c <name>` to keep new commits, or `git switch main`
`Permission denied (publickey)`	SSH key not offered or not registered	`ssh -vT git@<host>`; check `ssh-add -l`, the key on the server, the right account
`fatal: refusing to merge unrelated histories`	Two repos with no common commit	Usually a mistake (wrong remote); `--allow-unrelated-histories` only if intended
Whole files show as changed	Line-ending or file-mode differences	`.gitattributes` (`* text=auto`), `core.autocrlf`, `core.fileMode false` on some filesystems
`error: Your local changes would be overwritten`	Uncommitted changes conflict with the switch/pull	Commit, `git stash`, or `git restore` them
Repository is huge	Large files in history	`git count-objects -vH`, find big blobs, move to LFS; rewrite history only if it's worth the disruption

A .gitattributes file in every repository settles line endings for everyone:

* text=auto eol=lf
*.ps1 text eol=crlf
*.png binary
*.iso filter=lfs diff=lfs merge=lfs -text

Try it: speed and fixes

Clone a large public repository three ways (full, --depth 1, --filter=blob:none) and compare time and git count-objects -vH.
Use git sparse-checkout set to keep only one folder.
Reproduce a non-fast-forward rejection with two clones of a test repo, then fix it properly.
Check out a tag, make a commit in detached HEAD, and rescue it onto a branch.

Recap

Shallow for CI, partial + sparse for big monorepos, LFS or an artifact store for binaries.
Prefer versioned modules/packages over submodules and subtrees.
Know the classic errors: non-fast-forward, detached HEAD, publickey, line endings; add a .gitattributes.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.