how to keep git-fetch from running out of memory?

[Kartik Agaram]

I have a git repo with some large files that I'm no longer able to
update. git fetch keeps running out of memory:

fatal: Out of memory, malloc failed
fatal: unpack-objects died with error code 128

Anybody know how to keep it from compressing the refs into packfiles?
I've experimented with core.compression, pack.compression,
pack.windowMemory, pack.packSizeLimit, all without luck :(

[Shawn Pearce]

Post by Kartik Agaram
I have a git repo with some large files that I'm no longer able to
=A0fatal: Out of memory, malloc failed
=A0fatal: unpack-objects died with error code 128
Anybody know how to keep it from compressing the refs into packfiles?
I've experimented with core.compression, pack.compression,
pack.windowMemory, pack.packSizeLimit, all without luck :(

Instead of playing with these settings, try transfer.unpackLimit 1. It
will force the code to use index-pack rather than unpack-objects,
which has a different memory profile.

However, that may still be insufficient. A big object must still be
allocated in memory in order to compute its SHA-1. If you don't have
sufficient memory, you need to increase your ulimits, or reconfigure
your system to have more virtual memory available to the process.

--=20
Shawn.

[Junio C Hamano]

Post by Shawn Pearce
However, that may still be insufficient. A big object must still be
allocated in memory in order to compute its SHA-1.

I wonder if we can stream this?

[Shawn Pearce]

Post by Junio C Hamano

Post by Shawn Pearce
However, that may still be insufficient. A big object must still be
allocated in memory in order to compute its SHA-1.

I wonder if we can stream this?

If its a non-delta, yes. We already do this in JGit for blobs over
50M. (Trees, commits and tags are not streamed because we might need
to do fsck on them and the fsck code for these object types requires
the entire object in a single buffer.)

Its probably not a huge change to index-pack.c. But I haven't dug
around in there in a while. I wish I could say I will look at this on
Monday, but I don't have the time.