The blog about containerisation, virtual machines and useful shell snippets and findings

How to remove broken LFS assets from git repo.

Sometimes you can get error similar to this
Object does not exist on the server: [404] Object does not exist on the server

That means you messed with upload of your artifacts to LFS and they changed at some point and being commited to repo.
The only way to fix that is to remove these files from commit, that will overwrite your history, hense you have to push your branch with --force flag to overwrite your remote repository.

So here is your hard way to fix that:

  1. Search for the commit that contains your assets. In my case it is Runtime/platforms/hpux/hpux_rebuilt.tar
git log --raw --all --full-history -- Runtime/platforms/hpux/hpux_rebuilt.tar
tree 901cb036d4dfef5c36c66d6452b3d496d0e22772
parent 19848337754164ea17afcd1eb7bd55ef512b3c34
author Maksym Prokopov  1548405977 +0700
committer Maksym Prokopov  1548405977 +0700

added script clause

search your file with

git cat-file -p 901cb and drill down until you find your assed blob sha1
100644 blob 496ee2ca6a2f08396a4076fe43dedf3dc0da8b6d	.gitignore
040000 tree 1c93b94f6ec752528a46741a7ff94e81d46afdcf	AppServer
040000 tree b32efb40aa560188970fc4f470c697e06f7c7030	Build
100644 blob 050d454e84afc229f35144ef8f13087c58d11f8a	Jenkinsfile_AppServer_Cobol
040000 tree ceac52278585b596885410e38021bbcbb95cd7b6	Runtime
git cat-file -p ceac52
040000 tree ff05d245ea0782b43355affc5fe8237ec8619bdc	platforms
git cat-file -p ff05d245ea0782
040000 tree c9c55c219bc5cf5436cdb4d4dcf761a088677bc2	aix
040000 tree 65cc3c3899d9bfecbf95999fdaed24c2235fc887	hpux
040000 tree 13bada715fbde55b9d1e3be0aa0c82a8ed886ca0	intel_nt
040000 tree ad1b0ec603f2895a91a3241fa3ad034c97645b44	solaris
git cat-file -p 65cc3c3899d9bfecbf95999fda
040000 tree 3477a0392bd9733297c52284a85776bf2461f7d8	SDK
100644 blob ae6c4b1828104c732c36a680c5b298fb5769fbaf	cshp1_list
040000 tree 1bfe688c02bbf87836d70709318d750b89f17f6a	exec
100644 blob 808487ec8d13531604c58a632dec9373c2629647	executables_list
100644 blob f2033d65e09b598f6a19e141cae59c980f8595ae	hpux_rebuilt.tar
100644 blob 82cf5d66ffdecde3f059f9bc9914f1fc8498b5f0	non_executables_list
040000 tree 182465cb595fa768df9376f558110eff8cc88afc	sbin

here is your blob id

100644 blob f2033d65e09b598f6a19e141cae59c980f8595ae	hpux_rebuilt.tar
  1. Find assets blog_id. Put their blob_id to some file.

Put f2033d65e09b598f6a19e141cae59c980f8595ae to blob-ids file

Repeate step 1 for every file your’re missing.

  1. Download and execute (https://rtyley.github.io/bfg-repo-cleaner/ BFG utility) with flag -bi and your file with blob ids.
java -jar ~/Downloads/bfg-1.13.0.jar -bi blob-ids
  1. Revert last commit and remove files.
    You’ll see
Updating references:    100% (10/10)
...Ref update completed in 15 ms.

Commit Tree-Dirt History
------------------------

	Earliest                                              Latest
	|                                                          |
	DmDmDDmDmDmmDmDmDDDmmDmDmDmDDDmDmDmmDmDmDDDmDmDmDmDDmDDmmmDm

	D = dirty commits (file tree fixed)
	m = modified commits (commit message or parents changed)
	. = clean commits (no changes to file tree)

	                        Before     After
	-------------------------------------------
	First modified commit | 39384790 | 5c5a7a09
	Last dirty commit     | 71d09db3 | 495a1329

Deleted files
-------------

	Filename                  Git id
	------------------------------------------
	15schemas.tar           | b795d231 (134 B)
	ciar_vb4_15.tar         | 2d06bbb5 (133 B)
	ciar_xsl_15.tar         | 26c79c30 (132 B)
	civt_vb4_15.tar         | 23d54549 (132 B)
	civt_xsl_15.tar         | 7201bc36 (131 B)
	mysql.tar               | e3d9d62d (131 B)
	wrunora.15.1.0.0_11.2.0 | d752af96 (133 B)

In total, 109 object ids were changed. Full details are logged here:

  1. Redo commit.
  2. Execute git garbage collection.
git reflog expire --expire=now --all && git gc --prune=now --aggressive
  1. Push your changes with --force to remote repo.
git push origin --force

If you want to re-add LFS files
you can do it with

git lfs track "filename"