technical/pack-heuristics.html

   1 <!DOCTYPE html>
   2 <html xmlns="http://www.w3.org/1999/xhtml" lang="en">
   3 <head>
   4 <meta charset="UTF-8"/>
   5 <meta http-equiv="X-UA-Compatible" content="IE=edge"/>
   6 <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
   7 <meta name="generator" content="Asciidoctor 2.0.20"/>
   8 <title>Concerning Git&#8217;s Packing Heuristics</title>
   9 <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700"/>
  10 <style>
  11 /*! Asciidoctor default stylesheet | MIT License | https://asciidoctor.org */
  12 /* Uncomment the following line when using as a custom stylesheet */
  13 /* @import "https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700"; */
  14 html{font-family:sans-serif;-webkit-text-size-adjust:100%}
  15 a{background:none}
  16 a:focus{outline:thin dotted}
  17 a:active,a:hover{outline:0}
  18 h1{font-size:2em;margin:.67em 0}
  19 b,strong{font-weight:bold}
  20 abbr{font-size:.9em}
  21 abbr[title]{cursor:help;border-bottom:1px dotted #dddddf;text-decoration:none}
  22 dfn{font-style:italic}
  23 hr{height:0}
  24 mark{background:#ff0;color:#000}
  25 code,kbd,pre,samp{font-family:monospace;font-size:1em}
  26 pre{white-space:pre-wrap}
  27 q{quotes:"\201C" "\201D" "\2018" "\2019"}
  28 small{font-size:80%}
  29 sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}
  30 sup{top:-.5em}
  31 sub{bottom:-.25em}
  32 img{border:0}
  33 svg:not(:root){overflow:hidden}
  34 figure{margin:0}
  35 audio,video{display:inline-block}
  36 audio:not([controls]){display:none;height:0}
  37 fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}
  38 legend{border:0;padding:0}
  39 button,input,select,textarea{font-family:inherit;font-size:100%;margin:0}
  40 button,input{line-height:normal}
  41 button,select{text-transform:none}
  42 button,html input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer}
  43 button[disabled],html input[disabled]{cursor:default}
  44 input[type=checkbox],input[type=radio]{padding:0}
  45 button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}
  46 textarea{overflow:auto;vertical-align:top}
  47 table{border-collapse:collapse;border-spacing:0}
  48 *,::before,::after{box-sizing:border-box}
  49 html,body{font-size:100%}
  50 body{background:#fff;color:rgba(0,0,0,.8);padding:0;margin:0;font-family:"Noto Serif","DejaVu Serif",serif;line-height:1;position:relative;cursor:auto;-moz-tab-size:4;-o-tab-size:4;tab-size:4;word-wrap:anywhere;-moz-osx-font-smoothing:grayscale;-webkit-font-smoothing:antialiased}
  51 a:hover{cursor:pointer}
  52 img,object,embed{max-width:100%;height:auto}
  53 object,embed{height:100%}
  54 img{-ms-interpolation-mode:bicubic}
  55 .left{float:left!important}
  56 .right{float:right!important}
  57 .text-left{text-align:left!important}
  58 .text-right{text-align:right!important}
  59 .text-center{text-align:center!important}
  60 .text-justify{text-align:justify!important}
  61 .hide{display:none}
  62 img,object,svg{display:inline-block;vertical-align:middle}
  63 textarea{height:auto;min-height:50px}
  64 select{width:100%}
  65 .subheader,.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{line-height:1.45;color:#7a2518;font-weight:400;margin-top:0;margin-bottom:.25em}
  66 div,dl,dt,dd,ul,ol,li,h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0}
  67 a{color:#2156a5;text-decoration:underline;line-height:inherit}
  68 a:hover,a:focus{color:#1d4b8f}
  69 a img{border:0}
  70 p{line-height:1.6;margin-bottom:1.25em;text-rendering:optimizeLegibility}
  71 p aside{font-size:.875em;line-height:1.35;font-style:italic}
  72 h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{font-family:"Open Sans","DejaVu Sans",sans-serif;font-weight:300;font-style:normal;color:#ba3925;text-rendering:optimizeLegibility;margin-top:1em;margin-bottom:.5em;line-height:1.0125em}
  73 h1 small,h2 small,h3 small,#toctitle small,.sidebarblock>.content>.title small,h4 small,h5 small,h6 small{font-size:60%;color:#e99b8f;line-height:0}
  74 h1{font-size:2.125em}
  75 h2{font-size:1.6875em}
  76 h3,#toctitle,.sidebarblock>.content>.title{font-size:1.375em}
  77 h4,h5{font-size:1.125em}
  78 h6{font-size:1em}
  79 hr{border:solid #dddddf;border-width:1px 0 0;clear:both;margin:1.25em 0 1.1875em}
  80 em,i{font-style:italic;line-height:inherit}
  81 strong,b{font-weight:bold;line-height:inherit}
  82 small{font-size:60%;line-height:inherit}
  83 code{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;color:rgba(0,0,0,.9)}
  84 ul,ol,dl{line-height:1.6;margin-bottom:1.25em;list-style-position:outside;font-family:inherit}
  85 ul,ol{margin-left:1.5em}
  86 ul li ul,ul li ol{margin-left:1.25em;margin-bottom:0}
  87 ul.circle{list-style-type:circle}
  88 ul.disc{list-style-type:disc}
  89 ul.square{list-style-type:square}
  90 ul.circle ul:not([class]),ul.disc ul:not([class]),ul.square ul:not([class]){list-style:inherit}
  91 ol li ul,ol li ol{margin-left:1.25em;margin-bottom:0}
  92 dl dt{margin-bottom:.3125em;font-weight:bold}
  93 dl dd{margin-bottom:1.25em}
  94 blockquote{margin:0 0 1.25em;padding:.5625em 1.25em 0 1.1875em;border-left:1px solid #ddd}
  95 blockquote,blockquote p{line-height:1.6;color:rgba(0,0,0,.85)}
  96 @media screen and (min-width:768px){h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2}
  97 h1{font-size:2.75em}
  98 h2{font-size:2.3125em}
  99 h3,#toctitle,.sidebarblock>.content>.title{font-size:1.6875em}
 100 h4{font-size:1.4375em}}
 101 table{background:#fff;margin-bottom:1.25em;border:1px solid #dedede;word-wrap:normal}
 102 table thead,table tfoot{background:#f7f8f7}
 103 table thead tr th,table thead tr td,table tfoot tr th,table tfoot tr td{padding:.5em .625em .625em;font-size:inherit;color:rgba(0,0,0,.8);text-align:left}
 104 table tr th,table tr td{padding:.5625em .625em;font-size:inherit;color:rgba(0,0,0,.8)}
 105 table tr.even,table tr.alt{background:#f8f8f7}
 106 table thead tr th,table tfoot tr th,table tbody tr td,table tr td,table tfoot tr td{line-height:1.6}
 107 h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2;word-spacing:-.05em}
 108 h1 strong,h2 strong,h3 strong,#toctitle strong,.sidebarblock>.content>.title strong,h4 strong,h5 strong,h6 strong{font-weight:400}
 109 .center{margin-left:auto;margin-right:auto}
 110 .stretch{width:100%}
 111 .clearfix::before,.clearfix::after,.float-group::before,.float-group::after{content:" ";display:table}
 112 .clearfix::after,.float-group::after{clear:both}
 113 :not(pre).nobreak{word-wrap:normal}
 114 :not(pre).nowrap{white-space:nowrap}
 115 :not(pre).pre-wrap{white-space:pre-wrap}
 116 :not(pre):not([class^=L])>code{font-size:.9375em;font-style:normal!important;letter-spacing:0;padding:.1em .5ex;word-spacing:-.15em;background:#f7f7f8;border-radius:4px;line-height:1.45;text-rendering:optimizeSpeed}
 117 pre{color:rgba(0,0,0,.9);font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;line-height:1.45;text-rendering:optimizeSpeed}
 118 pre code,pre pre{color:inherit;font-size:inherit;line-height:inherit}
 119 pre>code{display:block}
 120 pre.nowrap,pre.nowrap pre{white-space:pre;word-wrap:normal}
 121 em em{font-style:normal}
 122 strong strong{font-weight:400}
 123 .keyseq{color:rgba(51,51,51,.8)}
 124 kbd{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;display:inline-block;color:rgba(0,0,0,.8);font-size:.65em;line-height:1.45;background:#f7f7f7;border:1px solid #ccc;border-radius:3px;box-shadow:0 1px 0 rgba(0,0,0,.2),inset 0 0 0 .1em #fff;margin:0 .15em;padding:.2em .5em;vertical-align:middle;position:relative;top:-.1em;white-space:nowrap}
 125 .keyseq kbd:first-child{margin-left:0}
 126 .keyseq kbd:last-child{margin-right:0}
 127 .menuseq,.menuref{color:#000}
 128 .menuseq b:not(.caret),.menuref{font-weight:inherit}
 129 .menuseq{word-spacing:-.02em}
 130 .menuseq b.caret{font-size:1.25em;line-height:.8}
 131 .menuseq i.caret{font-weight:bold;text-align:center;width:.45em}
 132 b.button::before,b.button::after{position:relative;top:-1px;font-weight:400}
 133 b.button::before{content:"[";padding:0 3px 0 2px}
 134 b.button::after{content:"]";padding:0 2px 0 3px}
 135 p a>code:hover{color:rgba(0,0,0,.9)}
 136 #header,#content,#footnotes,#footer{width:100%;margin:0 auto;max-width:62.5em;*zoom:1;position:relative;padding-left:.9375em;padding-right:.9375em}
 137 #header::before,#header::after,#content::before,#content::after,#footnotes::before,#footnotes::after,#footer::before,#footer::after{content:" ";display:table}
 138 #header::after,#content::after,#footnotes::after,#footer::after{clear:both}
 139 #content{margin-top:1.25em}
 140 #content::before{content:none}
 141 #header>h1:first-child{color:rgba(0,0,0,.85);margin-top:2.25rem;margin-bottom:0}
 142 #header>h1:first-child+#toc{margin-top:8px;border-top:1px solid #dddddf}
 143 #header>h1:only-child,body.toc2 #header>h1:nth-last-child(2){border-bottom:1px solid #dddddf;padding-bottom:8px}
 144 #header .details{border-bottom:1px solid #dddddf;line-height:1.45;padding-top:.25em;padding-bottom:.25em;padding-left:.25em;color:rgba(0,0,0,.6);display:flex;flex-flow:row wrap}
 145 #header .details span:first-child{margin-left:-.125em}
 146 #header .details span.email a{color:rgba(0,0,0,.85)}
 147 #header .details br{display:none}
 148 #header .details br+span::before{content:"\00a0\2013\00a0"}
 149 #header .details br+span.author::before{content:"\00a0\22c5\00a0";color:rgba(0,0,0,.85)}
 150 #header .details br+span#revremark::before{content:"\00a0|\00a0"}
 151 #header #revnumber{text-transform:capitalize}
 152 #header #revnumber::after{content:"\00a0"}
 153 #content>h1:first-child:not([class]){color:rgba(0,0,0,.85);border-bottom:1px solid #dddddf;padding-bottom:8px;margin-top:0;padding-top:1rem;margin-bottom:1.25rem}
 154 #toc{border-bottom:1px solid #e7e7e9;padding-bottom:.5em}
 155 #toc>ul{margin-left:.125em}
 156 #toc ul.sectlevel0>li>a{font-style:italic}
 157 #toc ul.sectlevel0 ul.sectlevel1{margin:.5em 0}
 158 #toc ul{font-family:"Open Sans","DejaVu Sans",sans-serif;list-style-type:none}
 159 #toc li{line-height:1.3334;margin-top:.3334em}
 160 #toc a{text-decoration:none}
 161 #toc a:active{text-decoration:underline}
 162 #toctitle{color:#7a2518;font-size:1.2em}
 163 @media screen and (min-width:768px){#toctitle{font-size:1.375em}
 164 body.toc2{padding-left:15em;padding-right:0}
 165 #toc.toc2{margin-top:0!important;background:#f8f8f7;position:fixed;width:15em;left:0;top:0;border-right:1px solid #e7e7e9;border-top-width:0!important;border-bottom-width:0!important;z-index:1000;padding:1.25em 1em;height:100%;overflow:auto}
 166 #toc.toc2 #toctitle{margin-top:0;margin-bottom:.8rem;font-size:1.2em}
 167 #toc.toc2>ul{font-size:.9em;margin-bottom:0}
 168 #toc.toc2 ul ul{margin-left:0;padding-left:1em}
 169 #toc.toc2 ul.sectlevel0 ul.sectlevel1{padding-left:0;margin-top:.5em;margin-bottom:.5em}
 170 body.toc2.toc-right{padding-left:0;padding-right:15em}
 171 body.toc2.toc-right #toc.toc2{border-right-width:0;border-left:1px solid #e7e7e9;left:auto;right:0}}
 172 @media screen and (min-width:1280px){body.toc2{padding-left:20em;padding-right:0}
 173 #toc.toc2{width:20em}
 174 #toc.toc2 #toctitle{font-size:1.375em}
 175 #toc.toc2>ul{font-size:.95em}
 176 #toc.toc2 ul ul{padding-left:1.25em}
 177 body.toc2.toc-right{padding-left:0;padding-right:20em}}
 178 #content #toc{border:1px solid #e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;border-radius:4px}
 179 #content #toc>:first-child{margin-top:0}
 180 #content #toc>:last-child{margin-bottom:0}
 181 #footer{max-width:none;background:rgba(0,0,0,.8);padding:1.25em}
 182 #footer-text{color:hsla(0,0%,100%,.8);line-height:1.44}
 183 #content{margin-bottom:.625em}
 184 .sect1{padding-bottom:.625em}
 185 @media screen and (min-width:768px){#content{margin-bottom:1.25em}
 186 .sect1{padding-bottom:1.25em}}
 187 .sect1:last-child{padding-bottom:0}
 188 .sect1+.sect1{border-top:1px solid #e7e7e9}
 189 #content h1>a.anchor,h2>a.anchor,h3>a.anchor,#toctitle>a.anchor,.sidebarblock>.content>.title>a.anchor,h4>a.anchor,h5>a.anchor,h6>a.anchor{position:absolute;z-index:1001;width:1.5ex;margin-left:-1.5ex;display:block;text-decoration:none!important;visibility:hidden;text-align:center;font-weight:400}
 190 #content h1>a.anchor::before,h2>a.anchor::before,h3>a.anchor::before,#toctitle>a.anchor::before,.sidebarblock>.content>.title>a.anchor::before,h4>a.anchor::before,h5>a.anchor::before,h6>a.anchor::before{content:"\00A7";font-size:.85em;display:block;padding-top:.1em}
 191 #content h1:hover>a.anchor,#content h1>a.anchor:hover,h2:hover>a.anchor,h2>a.anchor:hover,h3:hover>a.anchor,#toctitle:hover>a.anchor,.sidebarblock>.content>.title:hover>a.anchor,h3>a.anchor:hover,#toctitle>a.anchor:hover,.sidebarblock>.content>.title>a.anchor:hover,h4:hover>a.anchor,h4>a.anchor:hover,h5:hover>a.anchor,h5>a.anchor:hover,h6:hover>a.anchor,h6>a.anchor:hover{visibility:visible}
 192 #content h1>a.link,h2>a.link,h3>a.link,#toctitle>a.link,.sidebarblock>.content>.title>a.link,h4>a.link,h5>a.link,h6>a.link{color:#ba3925;text-decoration:none}
 193 #content h1>a.link:hover,h2>a.link:hover,h3>a.link:hover,#toctitle>a.link:hover,.sidebarblock>.content>.title>a.link:hover,h4>a.link:hover,h5>a.link:hover,h6>a.link:hover{color:#a53221}
 194 details,.audioblock,.imageblock,.literalblock,.listingblock,.stemblock,.videoblock{margin-bottom:1.25em}
 195 details{margin-left:1.25rem}
 196 details>summary{cursor:pointer;display:block;position:relative;line-height:1.6;margin-bottom:.625rem;outline:none;-webkit-tap-highlight-color:transparent}
 197 details>summary::-webkit-details-marker{display:none}
 198 details>summary::before{content:"";border:solid transparent;border-left:solid;border-width:.3em 0 .3em .5em;position:absolute;top:.5em;left:-1.25rem;transform:translateX(15%)}
 199 details[open]>summary::before{border:solid transparent;border-top:solid;border-width:.5em .3em 0;transform:translateY(15%)}
 200 details>summary::after{content:"";width:1.25rem;height:1em;position:absolute;top:.3em;left:-1.25rem}
 201 .admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{text-rendering:optimizeLegibility;text-align:left;font-family:"Noto Serif","DejaVu Serif",serif;font-size:1rem;font-style:italic}
 202 table.tableblock.fit-content>caption.title{white-space:nowrap;width:0}
 203 .paragraph.lead>p,#preamble>.sectionbody>[class=paragraph]:first-of-type p{font-size:1.21875em;line-height:1.6;color:rgba(0,0,0,.85)}
 204 .admonitionblock>table{border-collapse:separate;border:0;background:none;width:100%}
 205 .admonitionblock>table td.icon{text-align:center;width:80px}
 206 .admonitionblock>table td.icon img{max-width:none}
 207 .admonitionblock>table td.icon .title{font-weight:bold;font-family:"Open Sans","DejaVu Sans",sans-serif;text-transform:uppercase}
 208 .admonitionblock>table td.content{padding-left:1.125em;padding-right:1.25em;border-left:1px solid #dddddf;color:rgba(0,0,0,.6);word-wrap:anywhere}
 209 .admonitionblock>table td.content>:last-child>:last-child{margin-bottom:0}
 210 .exampleblock>.content{border:1px solid #e6e6e6;margin-bottom:1.25em;padding:1.25em;background:#fff;border-radius:4px}
 211 .sidebarblock{border:1px solid #dbdbd6;margin-bottom:1.25em;padding:1.25em;background:#f3f3f2;border-radius:4px}
 212 .sidebarblock>.content>.title{color:#7a2518;margin-top:0;text-align:center}
 213 .exampleblock>.content>:first-child,.sidebarblock>.content>:first-child{margin-top:0}
 214 .exampleblock>.content>:last-child,.exampleblock>.content>:last-child>:last-child,.exampleblock>.content .olist>ol>li:last-child>:last-child,.exampleblock>.content .ulist>ul>li:last-child>:last-child,.exampleblock>.content .qlist>ol>li:last-child>:last-child,.sidebarblock>.content>:last-child,.sidebarblock>.content>:last-child>:last-child,.sidebarblock>.content .olist>ol>li:last-child>:last-child,.sidebarblock>.content .ulist>ul>li:last-child>:last-child,.sidebarblock>.content .qlist>ol>li:last-child>:last-child{margin-bottom:0}
 215 .literalblock pre,.listingblock>.content>pre{border-radius:4px;overflow-x:auto;padding:1em;font-size:.8125em}
 216 @media screen and (min-width:768px){.literalblock pre,.listingblock>.content>pre{font-size:.90625em}}
 217 @media screen and (min-width:1280px){.literalblock pre,.listingblock>.content>pre{font-size:1em}}
 218 .literalblock pre,.listingblock>.content>pre:not(.highlight),.listingblock>.content>pre[class=highlight],.listingblock>.content>pre[class^="highlight "]{background:#f7f7f8}
 219 .literalblock.output pre{color:#f7f7f8;background:rgba(0,0,0,.9)}
 220 .listingblock>.content{position:relative}
 221 .listingblock code[data-lang]::before{display:none;content:attr(data-lang);position:absolute;font-size:.75em;top:.425rem;right:.5rem;line-height:1;text-transform:uppercase;color:inherit;opacity:.5}
 222 .listingblock:hover code[data-lang]::before{display:block}
 223 .listingblock.terminal pre .command::before{content:attr(data-prompt);padding-right:.5em;color:inherit;opacity:.5}
 224 .listingblock.terminal pre .command:not([data-prompt])::before{content:"$"}
 225 .listingblock pre.highlightjs{padding:0}
 226 .listingblock pre.highlightjs>code{padding:1em;border-radius:4px}
 227 .listingblock pre.prettyprint{border-width:0}
 228 .prettyprint{background:#f7f7f8}
 229 pre.prettyprint .linenums{line-height:1.45;margin-left:2em}
 230 pre.prettyprint li{background:none;list-style-type:inherit;padding-left:0}
 231 pre.prettyprint li code[data-lang]::before{opacity:1}
 232 pre.prettyprint li:not(:first-child) code[data-lang]::before{display:none}
 233 table.linenotable{border-collapse:separate;border:0;margin-bottom:0;background:none}
 234 table.linenotable td[class]{color:inherit;vertical-align:top;padding:0;line-height:inherit;white-space:normal}
 235 table.linenotable td.code{padding-left:.75em}
 236 table.linenotable td.linenos,pre.pygments .linenos{border-right:1px solid;opacity:.35;padding-right:.5em;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none}
 237 pre.pygments span.linenos{display:inline-block;margin-right:.75em}
 238 .quoteblock{margin:0 1em 1.25em 1.5em;display:table}
 239 .quoteblock:not(.excerpt)>.title{margin-left:-1.5em;margin-bottom:.75em}
 240 .quoteblock blockquote,.quoteblock p{color:rgba(0,0,0,.85);font-size:1.15rem;line-height:1.75;word-spacing:.1em;letter-spacing:0;font-style:italic;text-align:justify}
 241 .quoteblock blockquote{margin:0;padding:0;border:0}
 242 .quoteblock blockquote::before{content:"\201c";float:left;font-size:2.75em;font-weight:bold;line-height:.6em;margin-left:-.6em;color:#7a2518;text-shadow:0 1px 2px rgba(0,0,0,.1)}
 243 .quoteblock blockquote>.paragraph:last-child p{margin-bottom:0}
 244 .quoteblock .attribution{margin-top:.75em;margin-right:.5ex;text-align:right}
 245 .verseblock{margin:0 1em 1.25em}
 246 .verseblock pre{font-family:"Open Sans","DejaVu Sans",sans-serif;font-size:1.15rem;color:rgba(0,0,0,.85);font-weight:300;text-rendering:optimizeLegibility}
 247 .verseblock pre strong{font-weight:400}
 248 .verseblock .attribution{margin-top:1.25rem;margin-left:.5ex}
 249 .quoteblock .attribution,.verseblock .attribution{font-size:.9375em;line-height:1.45;font-style:italic}
 250 .quoteblock .attribution br,.verseblock .attribution br{display:none}
 251 .quoteblock .attribution cite,.verseblock .attribution cite{display:block;letter-spacing:-.025em;color:rgba(0,0,0,.6)}
 252 .quoteblock.abstract blockquote::before,.quoteblock.excerpt blockquote::before,.quoteblock .quoteblock blockquote::before{display:none}
 253 .quoteblock.abstract blockquote,.quoteblock.abstract p,.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{line-height:1.6;word-spacing:0}
 254 .quoteblock.abstract{margin:0 1em 1.25em;display:block}
 255 .quoteblock.abstract>.title{margin:0 0 .375em;font-size:1.15em;text-align:center}
 256 .quoteblock.excerpt>blockquote,.quoteblock .quoteblock{padding:0 0 .25em 1em;border-left:.25em solid #dddddf}
 257 .quoteblock.excerpt,.quoteblock .quoteblock{margin-left:0}
 258 .quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{color:inherit;font-size:1.0625rem}
 259 .quoteblock.excerpt .attribution,.quoteblock .quoteblock .attribution{color:inherit;font-size:.85rem;text-align:left;margin-right:0}
 260 p.tableblock:last-child{margin-bottom:0}
 261 td.tableblock>.content{margin-bottom:1.25em;word-wrap:anywhere}
 262 td.tableblock>.content>:last-child{margin-bottom:-1.25em}
 263 table.tableblock,th.tableblock,td.tableblock{border:0 solid #dedede}
 264 table.grid-all>*>tr>*{border-width:1px}
 265 table.grid-cols>*>tr>*{border-width:0 1px}
 266 table.grid-rows>*>tr>*{border-width:1px 0}
 267 table.frame-all{border-width:1px}
 268 table.frame-ends{border-width:1px 0}
 269 table.frame-sides{border-width:0 1px}
 270 table.frame-none>colgroup+*>:first-child>*,table.frame-sides>colgroup+*>:first-child>*{border-top-width:0}
 271 table.frame-none>:last-child>:last-child>*,table.frame-sides>:last-child>:last-child>*{border-bottom-width:0}
 272 table.frame-none>*>tr>:first-child,table.frame-ends>*>tr>:first-child{border-left-width:0}
 273 table.frame-none>*>tr>:last-child,table.frame-ends>*>tr>:last-child{border-right-width:0}
 274 table.stripes-all>*>tr,table.stripes-odd>*>tr:nth-of-type(odd),table.stripes-even>*>tr:nth-of-type(even),table.stripes-hover>*>tr:hover{background:#f8f8f7}
 275 th.halign-left,td.halign-left{text-align:left}
 276 th.halign-right,td.halign-right{text-align:right}
 277 th.halign-center,td.halign-center{text-align:center}
 278 th.valign-top,td.valign-top{vertical-align:top}
 279 th.valign-bottom,td.valign-bottom{vertical-align:bottom}
 280 th.valign-middle,td.valign-middle{vertical-align:middle}
 281 table thead th,table tfoot th{font-weight:bold}
 282 tbody tr th{background:#f7f8f7}
 283 tbody tr th,tbody tr th p,tfoot tr th,tfoot tr th p{color:rgba(0,0,0,.8);font-weight:bold}
 284 p.tableblock>code:only-child{background:none;padding:0}
 285 p.tableblock{font-size:1em}
 286 ol{margin-left:1.75em}
 287 ul li ol{margin-left:1.5em}
 288 dl dd{margin-left:1.125em}
 289 dl dd:last-child,dl dd:last-child>:last-child{margin-bottom:0}
 290 li p,ul dd,ol dd,.olist .olist,.ulist .ulist,.ulist .olist,.olist .ulist{margin-bottom:.625em}
 291 ul.checklist,ul.none,ol.none,ul.no-bullet,ol.no-bullet,ol.unnumbered,ul.unstyled,ol.unstyled{list-style-type:none}
 292 ul.no-bullet,ol.no-bullet,ol.unnumbered{margin-left:.625em}
 293 ul.unstyled,ol.unstyled{margin-left:0}
 294 li>p:empty:only-child::before{content:"";display:inline-block}
 295 ul.checklist>li>p:first-child{margin-left:-1em}
 296 ul.checklist>li>p:first-child>.fa-square-o:first-child,ul.checklist>li>p:first-child>.fa-check-square-o:first-child{width:1.25em;font-size:.8em;position:relative;bottom:.125em}
 297 ul.checklist>li>p:first-child>input[type=checkbox]:first-child{margin-right:.25em}
 298 ul.inline{display:flex;flex-flow:row wrap;list-style:none;margin:0 0 .625em -1.25em}
 299 ul.inline>li{margin-left:1.25em}
 300 .unstyled dl dt{font-weight:400;font-style:normal}
 301 ol.arabic{list-style-type:decimal}
 302 ol.decimal{list-style-type:decimal-leading-zero}
 303 ol.loweralpha{list-style-type:lower-alpha}
 304 ol.upperalpha{list-style-type:upper-alpha}
 305 ol.lowerroman{list-style-type:lower-roman}
 306 ol.upperroman{list-style-type:upper-roman}
 307 ol.lowergreek{list-style-type:lower-greek}
 308 .hdlist>table,.colist>table{border:0;background:none}
 309 .hdlist>table>tbody>tr,.colist>table>tbody>tr{background:none}
 310 td.hdlist1,td.hdlist2{vertical-align:top;padding:0 .625em}
 311 td.hdlist1{font-weight:bold;padding-bottom:1.25em}
 312 td.hdlist2{word-wrap:anywhere}
 313 .literalblock+.colist,.listingblock+.colist{margin-top:-.5em}
 314 .colist td:not([class]):first-child{padding:.4em .75em 0;line-height:1;vertical-align:top}
 315 .colist td:not([class]):first-child img{max-width:none}
 316 .colist td:not([class]):last-child{padding:.25em 0}
 317 .thumb,.th{line-height:0;display:inline-block;border:4px solid #fff;box-shadow:0 0 0 1px #ddd}
 318 .imageblock.left{margin:.25em .625em 1.25em 0}
 319 .imageblock.right{margin:.25em 0 1.25em .625em}
 320 .imageblock>.title{margin-bottom:0}
 321 .imageblock.thumb,.imageblock.th{border-width:6px}
 322 .imageblock.thumb>.title,.imageblock.th>.title{padding:0 .125em}
 323 .image.left,.image.right{margin-top:.25em;margin-bottom:.25em;display:inline-block;line-height:0}
 324 .image.left{margin-right:.625em}
 325 .image.right{margin-left:.625em}
 326 a.image{text-decoration:none;display:inline-block}
 327 a.image object{pointer-events:none}
 328 sup.footnote,sup.footnoteref{font-size:.875em;position:static;vertical-align:super}
 329 sup.footnote a,sup.footnoteref a{text-decoration:none}
 330 sup.footnote a:active,sup.footnoteref a:active{text-decoration:underline}
 331 #footnotes{padding-top:.75em;padding-bottom:.75em;margin-bottom:.625em}
 332 #footnotes hr{width:20%;min-width:6.25em;margin:-.25em 0 .75em;border-width:1px 0 0}
 333 #footnotes .footnote{padding:0 .375em 0 .225em;line-height:1.3334;font-size:.875em;margin-left:1.2em;margin-bottom:.2em}
 334 #footnotes .footnote a:first-of-type{font-weight:bold;text-decoration:none;margin-left:-1.05em}
 335 #footnotes .footnote:last-of-type{margin-bottom:0}
 336 #content #footnotes{margin-top:-.625em;margin-bottom:0;padding:.75em 0}
 337 div.unbreakable{page-break-inside:avoid}
 338 .big{font-size:larger}
 339 .small{font-size:smaller}
 340 .underline{text-decoration:underline}
 341 .overline{text-decoration:overline}
 342 .line-through{text-decoration:line-through}
 343 .aqua{color:#00bfbf}
 344 .aqua-background{background:#00fafa}
 345 .black{color:#000}
 346 .black-background{background:#000}
 347 .blue{color:#0000bf}
 348 .blue-background{background:#0000fa}
 349 .fuchsia{color:#bf00bf}
 350 .fuchsia-background{background:#fa00fa}
 351 .gray{color:#606060}
 352 .gray-background{background:#7d7d7d}
 353 .green{color:#006000}
 354 .green-background{background:#007d00}
 355 .lime{color:#00bf00}
 356 .lime-background{background:#00fa00}
 357 .maroon{color:#600000}
 358 .maroon-background{background:#7d0000}
 359 .navy{color:#000060}
 360 .navy-background{background:#00007d}
 361 .olive{color:#606000}
 362 .olive-background{background:#7d7d00}
 363 .purple{color:#600060}
 364 .purple-background{background:#7d007d}
 365 .red{color:#bf0000}
 366 .red-background{background:#fa0000}
 367 .silver{color:#909090}
 368 .silver-background{background:#bcbcbc}
 369 .teal{color:#006060}
 370 .teal-background{background:#007d7d}
 371 .white{color:#bfbfbf}
 372 .white-background{background:#fafafa}
 373 .yellow{color:#bfbf00}
 374 .yellow-background{background:#fafa00}
 375 span.icon>.fa{cursor:default}
 376 a span.icon>.fa{cursor:inherit}
 377 .admonitionblock td.icon [class^="fa icon-"]{font-size:2.5em;text-shadow:1px 1px 2px rgba(0,0,0,.5);cursor:default}
 378 .admonitionblock td.icon .icon-note::before{content:"\f05a";color:#19407c}
 379 .admonitionblock td.icon .icon-tip::before{content:"\f0eb";text-shadow:1px 1px 2px rgba(155,155,0,.8);color:#111}
 380 .admonitionblock td.icon .icon-warning::before{content:"\f071";color:#bf6900}
 381 .admonitionblock td.icon .icon-caution::before{content:"\f06d";color:#bf3400}
 382 .admonitionblock td.icon .icon-important::before{content:"\f06a";color:#bf0000}
 383 .conum[data-value]{display:inline-block;color:#fff!important;background:rgba(0,0,0,.8);border-radius:50%;text-align:center;font-size:.75em;width:1.67em;height:1.67em;line-height:1.67em;font-family:"Open Sans","DejaVu Sans",sans-serif;font-style:normal;font-weight:bold}
 384 .conum[data-value] *{color:#fff!important}
 385 .conum[data-value]+b{display:none}
 386 .conum[data-value]::after{content:attr(data-value)}
 387 pre .conum[data-value]{position:relative;top:-.125em}
 388 b.conum *{color:inherit!important}
 389 .conum:not([data-value]):empty{display:none}
 390 dt,th.tableblock,td.content,div.footnote{text-rendering:optimizeLegibility}
 391 h1,h2,p,td.content,span.alt,summary{letter-spacing:-.01em}
 392 p strong,td.content strong,div.footnote strong{letter-spacing:-.005em}
 393 p,blockquote,dt,td.content,td.hdlist1,span.alt,summary{font-size:1.0625rem}
 394 p{margin-bottom:1.25rem}
 395 .sidebarblock p,.sidebarblock dt,.sidebarblock td.content,p.tableblock{font-size:1em}
 396 .exampleblock>.content{background:#fffef7;border-color:#e0e0dc;box-shadow:0 1px 4px #e0e0dc}
 397 .print-only{display:none!important}
 398 @page{margin:1.25cm .75cm}
 399 @media print{*{box-shadow:none!important;text-shadow:none!important}
 400 html{font-size:80%}
 401 a{color:inherit!important;text-decoration:underline!important}
 402 a.bare,a[href^="#"],a[href^="mailto:"]{text-decoration:none!important}
 403 a[href^="http:"]:not(.bare)::after,a[href^="https:"]:not(.bare)::after{content:"(" attr(href) ")";display:inline-block;font-size:.875em;padding-left:.25em}
 404 abbr[title]{border-bottom:1px dotted}
 405 abbr[title]::after{content:" (" attr(title) ")"}
 406 pre,blockquote,tr,img,object,svg{page-break-inside:avoid}
 407 thead{display:table-header-group}
 408 svg{max-width:100%}
 409 p,blockquote,dt,td.content{font-size:1em;orphans:3;widows:3}
 410 h2,h3,#toctitle,.sidebarblock>.content>.title{page-break-after:avoid}
 411 #header,#content,#footnotes,#footer{max-width:none}
 412 #toc,.sidebarblock,.exampleblock>.content{background:none!important}
 413 #toc{border-bottom:1px solid #dddddf!important;padding-bottom:0!important}
 414 body.book #header{text-align:center}
 415 body.book #header>h1:first-child{border:0!important;margin:2.5em 0 1em}
 416 body.book #header .details{border:0!important;display:block;padding:0!important}
 417 body.book #header .details span:first-child{margin-left:0!important}
 418 body.book #header .details br{display:block}
 419 body.book #header .details br+span::before{content:none!important}
 420 body.book #toc{border:0!important;text-align:left!important;padding:0!important;margin:0!important}
 421 body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-break-before:always}
 422 .listingblock code[data-lang]::before{display:block}
 423 #footer{padding:0 .9375em}
 424 .hide-on-print{display:none!important}
 425 .print-only{display:block!important}
 426 .hide-for-print{display:none!important}
 427 .show-for-print{display:inherit!important}}
 428 @media amzn-kf8,print{#header>h1:first-child{margin-top:1.25rem}
 429 .sect1{padding:0!important}
 430 .sect1+.sect1{border:0}
 431 #footer{background:none}
 432 #footer-text{color:rgba(0,0,0,.6);font-size:.9em}}
 433 @media amzn-kf8{#header,#content,#footnotes,#footer{padding:0}}
 434 </style>
 435 </head>
 436 <body class="article">
 437 <div id="header">
 438 <h1>Concerning Git&#8217;s Packing Heuristics</h1>
 439 <div class="details">
 440 <span id="revdate">2024-11-01</span>
 441 </div>
 442 </div>
 443 <div id="content">
 444 <div class="literalblock">
 445 <div class="content">
 446 <pre>Oh, here's a really stupid question:</pre>
 447 </div>
 448 </div>
 449 <div class="literalblock">
 450 <div class="content">
 451 <pre>             Where do I go
 452           to learn the details
 453 of Git's packing heuristics?</pre>
 454 </div>
 455 </div>
 456 <div class="paragraph">
 457 <p>Be careful what you ask!</p>
 458 </div>
 459 <div class="paragraph">
 460 <p>Followers of the Git, please open the Git IRC Log and turn to
 461 February 10, 2006.</p>
 462 </div>
 463 <div class="paragraph">
 464 <p>It&#8217;s a rare occasion, and we are joined by the King Git Himself,
 465 Linus Torvalds (linus).  Nathaniel Smith, (njs`), has the floor
 466 and seeks enlightenment.  Others are present, but silent.</p>
 467 </div>
 468 <div class="paragraph">
 469 <p>Let&#8217;s listen in!</p>
 470 </div>
 471 <div class="literalblock">
 472 <div class="content">
 473 <pre>   &lt;njs`&gt; Oh, here's a really stupid question -- where do I go to
 474 learn the details of Git's packing heuristics?  google avails
 475        me not, reading the source didn't help a lot, and wading
 476        through the whole mailing list seems less efficient than any
 477        of that.</pre>
 478 </div>
 479 </div>
 480 <div class="paragraph">
 481 <p>It is a bold start!  A plea for help combined with a simultaneous
 482 tri-part attack on some of the tried and true mainstays in the quest
 483 for enlightenment.  Brash accusations of google being useless. Hubris!
 484 Maligning the source.  Heresy!  Disdain for the mailing list archives.
 485 Woe.</p>
 486 </div>
 487 <div class="literalblock">
 488 <div class="content">
 489 <pre>&lt;pasky&gt; yes, the packing-related delta stuff is somewhat
 490     mysterious even for me ;)</pre>
 491 </div>
 492 </div>
 493 <div class="paragraph">
 494 <p>Ah!  Modesty after all.</p>
 495 </div>
 496 <div class="literalblock">
 497 <div class="content">
 498 <pre>  &lt;linus&gt; njs, I don't think the docs exist. That's something where
 499 I don't think anybody else than me even really got involved.
 500 Most of the rest of Git others have been busy with (especially
 501 Junio), but packing nobody touched after I did it.</pre>
 502 </div>
 503 </div>
 504 <div class="paragraph">
 505 <p>It&#8217;s cryptic, yet vague.  Linus in style for sure.  Wise men
 506 interpret this as an apology.  A few argue it is merely a
 507 statement of fact.</p>
 508 </div>
 509 <div class="literalblock">
 510 <div class="content">
 511 <pre>&lt;njs`&gt; I guess the next step is "read the source again", but I
 512     have to build up a certain level of gumption first :-)</pre>
 513 </div>
 514 </div>
 515 <div class="paragraph">
 516 <p>Indeed!  On both points.</p>
 517 </div>
 518 <div class="literalblock">
 519 <div class="content">
 520 <pre>&lt;linus&gt; The packing heuristic is actually really really simple.</pre>
 521 </div>
 522 </div>
 523 <div class="paragraph">
 524 <p>Bait&#8230;&#8203;</p>
 525 </div>
 526 <div class="literalblock">
 527 <div class="content">
 528 <pre>&lt;linus&gt; But strange.</pre>
 529 </div>
 530 </div>
 531 <div class="paragraph">
 532 <p>And switch.  That ought to do it!</p>
 533 </div>
 534 <div class="literalblock">
 535 <div class="content">
 536 <pre>&lt;linus&gt; Remember: Git really doesn't follow files. So what it does is
 537     - generate a list of all objects
 538     - sort the list according to magic heuristics
 539     - walk the list, using a sliding window, seeing if an object
 540       can be diffed against another object in the window
 541     - write out the list in recency order</pre>
 542 </div>
 543 </div>
 544 <div class="paragraph">
 545 <p>The traditional understatement:</p>
 546 </div>
 547 <div class="literalblock">
 548 <div class="content">
 549 <pre>&lt;njs`&gt; I suspect that what I'm missing is the precise definition of
 550     the word "magic"</pre>
 551 </div>
 552 </div>
 553 <div class="paragraph">
 554 <p>The traditional insight:</p>
 555 </div>
 556 <div class="literalblock">
 557 <div class="content">
 558 <pre>&lt;pasky&gt; yes</pre>
 559 </div>
 560 </div>
 561 <div class="paragraph">
 562 <p>And Babel-like confusion flowed.</p>
 563 </div>
 564 <div class="literalblock">
 565 <div class="content">
 566 <pre>&lt;njs`&gt; oh, hmm, and I'm not sure what this sliding window means either</pre>
 567 </div>
 568 </div>
 569 <div class="literalblock">
 570 <div class="content">
 571 <pre>&lt;pasky&gt; iirc, it appeared to me to be just the sha1 of the object
 572     when reading the code casually ...</pre>
 573 </div>
 574 </div>
 575 <div class="olist lowerroman">
 576 <ol class="lowerroman" type="i">
 577 <li>
 578 <p>which simply doesn&#8217;t sound as a very good heuristics, though ;)</p>
 579 <div class="literalblock">
 580 <div class="content">
 581 <pre>&lt;njs`&gt; .....and recency order.  okay, I think it's clear I didn't
 582    even realize how much I wasn't realizing :-)</pre>
 583 </div>
 584 </div>
 585 </li>
 586 </ol>
 587 </div>
 588 <div class="paragraph">
 589 <p>Ah, grasshopper!  And thus the enlightenment begins anew.</p>
 590 </div>
 591 <div class="literalblock">
 592 <div class="content">
 593 <pre>   &lt;linus&gt; The "magic" is actually in theory totally arbitrary.
 594        ANY order will give you a working pack, but no, it's not
 595 ordered by SHA-1.</pre>
 596 </div>
 597 </div>
 598 <div class="literalblock">
 599 <div class="content">
 600 <pre>Before talking about the ordering for the sliding delta
 601 window, let's talk about the recency order. That's more
 602 important in one way.</pre>
 603 </div>
 604 </div>
 605 <div class="literalblock">
 606 <div class="content">
 607 <pre>&lt;njs`&gt; Right, but if all you want is a working way to pack things
 608     together, you could just use cat and save yourself some
 609     trouble...</pre>
 610 </div>
 611 </div>
 612 <div class="paragraph">
 613 <p>Waaait for it&#8230;&#8203;.</p>
 614 </div>
 615 <div class="literalblock">
 616 <div class="content">
 617 <pre>&lt;linus&gt; The recency ordering (which is basically: put objects
 618     _physically_ into the pack in the order that they are
 619     "reachable" from the head) is important.</pre>
 620 </div>
 621 </div>
 622 <div class="literalblock">
 623 <div class="content">
 624 <pre>&lt;njs`&gt; okay</pre>
 625 </div>
 626 </div>
 627 <div class="literalblock">
 628 <div class="content">
 629 <pre>&lt;linus&gt; It's important because that's the thing that gives packs
 630     good locality. It keeps the objects close to the head (whether
 631     they are old or new, but they are _reachable_ from the head)
 632     at the head of the pack. So packs actually have absolutely
 633     _wonderful_ IO patterns.</pre>
 634 </div>
 635 </div>
 636 <div class="paragraph">
 637 <p>Read that again, because it is important.</p>
 638 </div>
 639 <div class="literalblock">
 640 <div class="content">
 641 <pre>&lt;linus&gt; But recency ordering is totally useless for deciding how
 642     to actually generate the deltas, so the delta ordering is
 643     something else.</pre>
 644 </div>
 645 </div>
 646 <div class="literalblock">
 647 <div class="content">
 648 <pre>The delta ordering is (wait for it):
 649 - first sort by the "basename" of the object, as defined by
 650   the name the object was _first_ reached through when
 651   generating the object list
 652 - within the same basename, sort by size of the object
 653 - but always sort different types separately (commits first).</pre>
 654 </div>
 655 </div>
 656 <div class="literalblock">
 657 <div class="content">
 658 <pre>That's not exactly it, but it's very close.</pre>
 659 </div>
 660 </div>
 661 <div class="literalblock">
 662 <div class="content">
 663 <pre>&lt;njs`&gt; The "_first_ reached" thing is not too important, just you
 664     need some way to break ties since the same objects may be
 665     reachable many ways, yes?</pre>
 666 </div>
 667 </div>
 668 <div class="paragraph">
 669 <p>And as if to clarify:</p>
 670 </div>
 671 <div class="literalblock">
 672 <div class="content">
 673 <pre>&lt;linus&gt; The point is that it's all really just any random
 674     heuristic, and the ordering is totally unimportant for
 675     correctness, but it helps a lot if the heuristic gives
 676     "clumping" for things that are likely to delta well against
 677     each other.</pre>
 678 </div>
 679 </div>
 680 <div class="paragraph">
 681 <p>It is an important point, so secretly, I did my own research and have
 682 included my results below.  To be fair, it has changed some over time.
 683 And through the magic of Revisionistic History, I draw upon this entry
 684 from The Git IRC Logs on my father&#8217;s birthday, March 1:</p>
 685 </div>
 686 <div class="literalblock">
 687 <div class="content">
 688 <pre> &lt;gitster&gt; The quote from the above linus should be rewritten a
 689      bit (wait for it):
 690      - first sort by type.  Different objects never delta with
 691 each other.
 692      - then sort by filename/dirname.  hash of the basename
 693        occupies the top BITS_PER_INT-DIR_BITS bits, and bottom
 694        DIR_BITS are for the hash of leading path elements.
 695      - then if we are doing "thin" pack, the objects we are _not_
 696        going to pack but we know about are sorted earlier than
 697        other objects.
 698      - and finally sort by size, larger to smaller.</pre>
 699 </div>
 700 </div>
 701 <div class="paragraph">
 702 <p>In one swell-foop, clarification and obscurification!  Nonetheless,
 703 authoritative.  Cryptic, yet concise.  It even solicits notions of
 704 quotes from The Source Code.  Clearly, more study is needed.</p>
 705 </div>
 706 <div class="literalblock">
 707 <div class="content">
 708 <pre>   &lt;gitster&gt; That's the sort order.  What this means is:
 709        - we do not delta different object types.
 710 - we prefer to delta the objects with the same full path, but
 711          allow files with the same name from different directories.
 712 - we always prefer to delta against objects we are not going
 713          to send, if there are some.
 714 - we prefer to delta against larger objects, so that we have
 715          lots of removals.</pre>
 716 </div>
 717 </div>
 718 <div class="literalblock">
 719 <div class="content">
 720 <pre>The penultimate rule is for "thin" packs.  It is used when
 721 the other side is known to have such objects.</pre>
 722 </div>
 723 </div>
 724 <div class="paragraph">
 725 <p>There it is again. "Thin" packs.  I&#8217;m thinking to myself, "What
 726 is a <em>thin</em> pack?"  So I ask:</p>
 727 </div>
 728 <div class="literalblock">
 729 <div class="content">
 730 <pre>&lt;jdl&gt; What is a "thin" pack?</pre>
 731 </div>
 732 </div>
 733 <div class="literalblock">
 734 <div class="content">
 735 <pre>&lt;gitster&gt; Use of --objects-edge to rev-list as the upstream of
 736     pack-objects.  The pack transfer protocol negotiates that.</pre>
 737 </div>
 738 </div>
 739 <div class="paragraph">
 740 <p>Woo hoo!  Cleared that <em>right</em> up!</p>
 741 </div>
 742 <div class="literalblock">
 743 <div class="content">
 744 <pre>&lt;gitster&gt; There are two directions - push and fetch.</pre>
 745 </div>
 746 </div>
 747 <div class="paragraph">
 748 <p>There!  Did you see it?  It is not <em>"push" and "pull"</em>!  How often the
 749 confusion has started here.  So casually mentioned, too!</p>
 750 </div>
 751 <div class="literalblock">
 752 <div class="content">
 753 <pre>&lt;gitster&gt; For push, git-send-pack invokes git-receive-pack on the
 754     other end.  The receive-pack says "I have up to these commits".
 755     send-pack looks at them, and computes what are missing from
 756     the other end.  So "thin" could be the default there.</pre>
 757 </div>
 758 </div>
 759 <div class="literalblock">
 760 <div class="content">
 761 <pre>       In the other direction, fetch, git-fetch-pack and
 762        git-clone-pack invokes git-upload-pack on the other end
 763 (via ssh or by talking to the daemon).</pre>
 764 </div>
 765 </div>
 766 <div class="literalblock">
 767 <div class="content">
 768 <pre>There are two cases: fetch-pack with -k and clone-pack is one,
 769        fetch-pack without -k is the other.  clone-pack and fetch-pack
 770        with -k will keep the downloaded packfile without expanded, so
 771        we do not use thin pack transfer.  Otherwise, the generated
 772        pack will have delta without base object in the same pack.</pre>
 773 </div>
 774 </div>
 775 <div class="literalblock">
 776 <div class="content">
 777 <pre>But fetch-pack without -k will explode the received pack into
 778 individual objects, so we automatically ask upload-pack to
 779 give us a thin pack if upload-pack supports it.</pre>
 780 </div>
 781 </div>
 782 <div class="paragraph">
 783 <p>OK then.</p>
 784 </div>
 785 <div class="paragraph">
 786 <p>Uh.</p>
 787 </div>
 788 <div class="paragraph">
 789 <p>Let&#8217;s return to the previous conversation still in progress.</p>
 790 </div>
 791 <div class="literalblock">
 792 <div class="content">
 793 <pre>&lt;njs`&gt; and "basename" means something like "the tail of end of
 794     path of file objects and dir objects, as per basename(3), and
 795     we just declare all commit and tag objects to have the same
 796     basename" or something?</pre>
 797 </div>
 798 </div>
 799 <div class="paragraph">
 800 <p>Luckily, that too is a point that gitster clarified for us!</p>
 801 </div>
 802 <div class="paragraph">
 803 <p>If I might add, the trick is to make files that <em>might</em> be similar be
 804 located close to each other in the hash buckets based on their file
 805 names.  It used to be that "foo/Makefile", "bar/baz/quux/Makefile" and
 806 "Makefile" all landed in the same bucket due to their common basename,
 807 "Makefile". However, now they land in "close" buckets.</p>
 808 </div>
 809 <div class="paragraph">
 810 <p>The algorithm allows not just for the <em>same</em> bucket, but for <em>close</em>
 811 buckets to be considered delta candidates.  The rationale is
 812 essentially that files, like Makefiles, often have very similar
 813 content no matter what directory they live in.</p>
 814 </div>
 815 <div class="literalblock">
 816 <div class="content">
 817 <pre>&lt;linus&gt; I played around with different delta algorithms, and with
 818     making the "delta window" bigger, but having too big of a
 819     sliding window makes it very expensive to generate the pack:
 820     you need to compare every object with a _ton_ of other objects.</pre>
 821 </div>
 822 </div>
 823 <div class="literalblock">
 824 <div class="content">
 825 <pre>There are a number of other trivial heuristics too, which
 826 basically boil down to "don't bother even trying to delta this
 827 pair" if we can tell before-hand that the delta isn't worth it
 828 (due to size differences, where we can take a previous delta
 829 result into account to decide that "ok, no point in trying
 830 that one, it will be worse").</pre>
 831 </div>
 832 </div>
 833 <div class="literalblock">
 834 <div class="content">
 835 <pre>End result: packing is actually very size efficient. It's
 836 somewhat CPU-wasteful, but on the other hand, since you're
 837 really only supposed to do it maybe once a month (and you can
 838 do it during the night), nobody really seems to care.</pre>
 839 </div>
 840 </div>
 841 <div class="paragraph">
 842 <p>Nice Engineering Touch, there.  Find when it doesn&#8217;t matter, and
 843 proclaim it a non-issue.  Good style too!</p>
 844 </div>
 845 <div class="literalblock">
 846 <div class="content">
 847 <pre>&lt;njs`&gt; So, just to repeat to see if I'm following, we start by
 848     getting a list of the objects we want to pack, we sort it by
 849     this heuristic (basically lexicographically on the tuple
 850     (type, basename, size)).</pre>
 851 </div>
 852 </div>
 853 <div class="literalblock">
 854 <div class="content">
 855 <pre>Then we walk through this list, and calculate a delta of
 856 each object against the last n (tunable parameter) objects,
 857 and pick the smallest of these deltas.</pre>
 858 </div>
 859 </div>
 860 <div class="paragraph">
 861 <p>Vastly simplified, but the essence is there!</p>
 862 </div>
 863 <div class="literalblock">
 864 <div class="content">
 865 <pre>&lt;linus&gt; Correct.</pre>
 866 </div>
 867 </div>
 868 <div class="literalblock">
 869 <div class="content">
 870 <pre>&lt;njs`&gt; And then once we have picked a delta or fulltext to
 871     represent each object, we re-sort by recency, and write them
 872     out in that order.</pre>
 873 </div>
 874 </div>
 875 <div class="literalblock">
 876 <div class="content">
 877 <pre>&lt;linus&gt; Yup. Some other small details:</pre>
 878 </div>
 879 </div>
 880 <div class="paragraph">
 881 <p>And of course there is the "Other Shoe" Factor too.</p>
 882 </div>
 883 <div class="literalblock">
 884 <div class="content">
 885 <pre>&lt;linus&gt; - We limit the delta depth to another magic value (right
 886     now both the window and delta depth magic values are just "10")</pre>
 887 </div>
 888 </div>
 889 <div class="literalblock">
 890 <div class="content">
 891 <pre>&lt;njs`&gt; Hrm, my intuition is that you'd end up with really _bad_ IO
 892     patterns, because the things you want are near by, but to
 893     actually reconstruct them you may have to jump all over in
 894     random ways.</pre>
 895 </div>
 896 </div>
 897 <div class="literalblock">
 898 <div class="content">
 899 <pre>&lt;linus&gt; - When we write out a delta, and we haven't yet written
 900     out the object it is a delta against, we write out the base
 901     object first.  And no, when we reconstruct them, we actually
 902     get nice IO patterns, because:
 903     - larger objects tend to be "more recent" (Linus' law: files grow)
 904     - we actively try to generate deltas from a larger object to a
 905       smaller one
 906     - this means that the top-of-tree very seldom has deltas
 907       (i.e. deltas in _practice_ are "backwards deltas")</pre>
 908 </div>
 909 </div>
 910 <div class="paragraph">
 911 <p>Again, we should reread that whole paragraph.  Not just because
 912 Linus has slipped Linus&#8217;s Law in there on us, but because it is
 913 important.  Let&#8217;s make sure we clarify some of the points here:</p>
 914 </div>
 915 <div class="literalblock">
 916 <div class="content">
 917 <pre>&lt;njs`&gt; So the point is just that in practice, delta order and
 918     recency order match each other quite well.</pre>
 919 </div>
 920 </div>
 921 <div class="literalblock">
 922 <div class="content">
 923 <pre>   &lt;linus&gt; Yes. There's another nice side to this (and yes, it was
 924 designed that way ;):
 925        - the reason we generate deltas against the larger object is
 926   actually a big space saver too!</pre>
 927 </div>
 928 </div>
 929 <div class="literalblock">
 930 <div class="content">
 931 <pre>&lt;njs`&gt; Hmm, but your last comment (if "we haven't yet written out
 932     the object it is a delta against, we write out the base object
 933     first"), seems like it would make these facts mostly
 934     irrelevant because even if in practice you would not have to
 935     wander around much, in fact you just brute-force say that in
 936     the cases where you might have to wander, don't do that :-)</pre>
 937 </div>
 938 </div>
 939 <div class="literalblock">
 940 <div class="content">
 941 <pre>&lt;linus&gt; Yes and no. Notice the rule: we only write out the base
 942     object first if the delta against it was more recent.  That
 943     means that you can actually have deltas that refer to a base
 944     object that is _not_ close to the delta object, but that only
 945     happens when the delta is needed to generate an _old_ object.</pre>
 946 </div>
 947 </div>
 948 <div class="literalblock">
 949 <div class="content">
 950 <pre>&lt;linus&gt; See?</pre>
 951 </div>
 952 </div>
 953 <div class="paragraph">
 954 <p>Yeah, no.  I missed that on the first two or three readings myself.</p>
 955 </div>
 956 <div class="literalblock">
 957 <div class="content">
 958 <pre>&lt;linus&gt; This keeps the front of the pack dense. The front of the
 959     pack never contains data that isn't relevant to a "recent"
 960     object.  The size optimization comes from our use of xdelta
 961     (but is true for many other delta algorithms): removing data
 962     is cheaper (in size) than adding data.</pre>
 963 </div>
 964 </div>
 965 <div class="literalblock">
 966 <div class="content">
 967 <pre>       When you remove data, you only need to say "copy bytes n--m".
 968 In contrast, in a delta that _adds_ data, you have to say "add
 969        these bytes: 'actual data goes here'"</pre>
 970 </div>
 971 </div>
 972 <div class="ulist">
 973 <ul>
 974 <li>
 975 <p>njs` has quit: Read error: 104 (Connection reset by peer)</p>
 976 <div class="literalblock">
 977 <div class="content">
 978 <pre>&lt;linus&gt; Uhhuh. I hope I didn't blow njs` mind.</pre>
 979 </div>
 980 </div>
 981 </li>
 982 <li>
 983 <p>njs` has joined channel #git</p>
 984 <div class="literalblock">
 985 <div class="content">
 986 <pre>&lt;pasky&gt; :)</pre>
 987 </div>
 988 </div>
 989 </li>
 990 </ul>
 991 </div>
 992 <div class="paragraph">
 993 <p>The silent observers are amused.  Of course.</p>
 994 </div>
 995 <div class="paragraph">
 996 <p>And as if njs` was expected to be omniscient:</p>
 997 </div>
 998 <div class="literalblock">
 999 <div class="content">
1000 <pre>&lt;linus&gt; njs - did you miss anything?</pre>
1001 </div>
1002 </div>
1003 <div class="paragraph">
1004 <p>OK, I&#8217;ll spell it out.  That&#8217;s Geek Humor.  If njs` was not actually
1005 connected for a little bit there, how would he know if missed anything
1006 while he was disconnected?  He&#8217;s a benevolent dictator with a sense of
1007 humor!  Well noted!</p>
1008 </div>
1009 <div class="literalblock">
1010 <div class="content">
1011 <pre>&lt;njs`&gt; Stupid router.  Or gremlins, or whatever.</pre>
1012 </div>
1013 </div>
1014 <div class="paragraph">
1015 <p>It&#8217;s a cheap shot at Cisco.  Take 'em when you can.</p>
1016 </div>
1017 <div class="literalblock">
1018 <div class="content">
1019 <pre>&lt;njs`&gt; Yes and no. Notice the rule: we only write out the base
1020     object first if the delta against it was more recent.</pre>
1021 </div>
1022 </div>
1023 <div class="literalblock">
1024 <div class="content">
1025 <pre>       I'm getting lost in all these orders, let me re-read :-)
1026 So the write-out order is from most recent to least recent?
1027        (Conceivably it could be the opposite way too, I'm not sure if
1028        we've said) though my connection back at home is logging, so I
1029        can just read what you said there :-)</pre>
1030 </div>
1031 </div>
1032 <div class="paragraph">
1033 <p>And for those of you paying attention, the Omniscient Trick has just
1034 been detailed!</p>
1035 </div>
1036 <div class="literalblock">
1037 <div class="content">
1038 <pre>&lt;linus&gt; Yes, we always write out most recent first</pre>
1039 </div>
1040 </div>
1041 <div class="literalblock">
1042 <div class="content">
1043 <pre>&lt;njs`&gt; And, yeah, I got the part about deeper-in-history stuff
1044     having worse IO characteristics, one sort of doesn't care.</pre>
1045 </div>
1046 </div>
1047 <div class="literalblock">
1048 <div class="content">
1049 <pre>&lt;linus&gt; With the caveat that if the "most recent" needs an older
1050     object to delta against (hey, shrinking sometimes does
1051     happen), we write out the old object with the delta.</pre>
1052 </div>
1053 </div>
1054 <div class="literalblock">
1055 <div class="content">
1056 <pre>&lt;njs`&gt; (if only it happened more...)</pre>
1057 </div>
1058 </div>
1059 <div class="literalblock">
1060 <div class="content">
1061 <pre>   &lt;linus&gt; Anyway, the pack-file could easily be denser still, but
1062 because it's used both for streaming (the Git protocol) and
1063        for on-disk, it has a few pessimizations.</pre>
1064 </div>
1065 </div>
1066 <div class="paragraph">
1067 <p>Actually, it is a made-up word. But it is a made-up word being
1068 used as setup for a later optimization, which is a real word:</p>
1069 </div>
1070 <div class="literalblock">
1071 <div class="content">
1072 <pre>&lt;linus&gt; In particular, while the pack-file is then compressed,
1073     it's compressed just one object at a time, so the actual
1074     compression factor is less than it could be in theory. But it
1075     means that it's all nice random-access with a simple index to
1076     do "object name-&gt;location in packfile" translation.</pre>
1077 </div>
1078 </div>
1079 <div class="literalblock">
1080 <div class="content">
1081 <pre>&lt;njs`&gt; I'm assuming the real win for delta-ing large-&gt;small is
1082     more homogeneous statistics for gzip to run over?</pre>
1083 </div>
1084 </div>
1085 <div class="literalblock">
1086 <div class="content">
1087 <pre>(You have to put the bytes in one place or another, but
1088 putting them in a larger blob wins on compression)</pre>
1089 </div>
1090 </div>
1091 <div class="literalblock">
1092 <div class="content">
1093 <pre>Actually, what is the compression strategy -- each delta
1094 individually gzipped, the whole file gzipped, somewhere in
1095 between, no compression at all, ....?</pre>
1096 </div>
1097 </div>
1098 <div class="literalblock">
1099 <div class="content">
1100 <pre>Right.</pre>
1101 </div>
1102 </div>
1103 <div class="paragraph">
1104 <p>Reality IRC sets in.  For example:</p>
1105 </div>
1106 <div class="literalblock">
1107 <div class="content">
1108 <pre>&lt;pasky&gt; I'll read the rest in the morning, I really have to go
1109     sleep or there's no hope whatsoever for me at the today's
1110     exam... g'nite all.</pre>
1111 </div>
1112 </div>
1113 <div class="paragraph">
1114 <p>Heh.</p>
1115 </div>
1116 <div class="literalblock">
1117 <div class="content">
1118 <pre>&lt;linus&gt; pasky: g'nite</pre>
1119 </div>
1120 </div>
1121 <div class="literalblock">
1122 <div class="content">
1123 <pre>&lt;njs`&gt; pasky: 'luck</pre>
1124 </div>
1125 </div>
1126 <div class="literalblock">
1127 <div class="content">
1128 <pre>&lt;linus&gt; Right: large-&gt;small matters exactly because of compression
1129     behaviour. If it was non-compressed, it probably wouldn't make
1130     any difference.</pre>
1131 </div>
1132 </div>
1133 <div class="literalblock">
1134 <div class="content">
1135 <pre>&lt;njs`&gt; yeah</pre>
1136 </div>
1137 </div>
1138 <div class="literalblock">
1139 <div class="content">
1140 <pre>&lt;linus&gt; Anyway: I'm not even trying to claim that the pack-files
1141     are perfect, but they do tend to have a nice balance of
1142     density vs ease-of use.</pre>
1143 </div>
1144 </div>
1145 <div class="paragraph">
1146 <p>Gasp!  OK, saved.  That&#8217;s a fair Engineering trade off.  Close call!
1147 In fact, Linus reflects on some Basic Engineering Fundamentals,
1148 design options, etc.</p>
1149 </div>
1150 <div class="literalblock">
1151 <div class="content">
1152 <pre>&lt;linus&gt; More importantly, they allow Git to still _conceptually_
1153     never deal with deltas at all, and be a "whole object" store.</pre>
1154 </div>
1155 </div>
1156 <div class="literalblock">
1157 <div class="content">
1158 <pre>       Which has some problems (we discussed bad huge-file
1159 behaviour on the Git lists the other day), but it does mean
1160 that the basic Git concepts are really really simple and
1161        straightforward.</pre>
1162 </div>
1163 </div>
1164 <div class="literalblock">
1165 <div class="content">
1166 <pre>It's all been quite stable.</pre>
1167 </div>
1168 </div>
1169 <div class="literalblock">
1170 <div class="content">
1171 <pre>Which I think is very much a result of having very simple
1172 basic ideas, so that there's never any confusion about what's
1173 going on.</pre>
1174 </div>
1175 </div>
1176 <div class="literalblock">
1177 <div class="content">
1178 <pre>Bugs happen, but they are "simple" bugs. And bugs that
1179 actually get some object store detail wrong are almost always
1180 so obvious that they never go anywhere.</pre>
1181 </div>
1182 </div>
1183 <div class="literalblock">
1184 <div class="content">
1185 <pre>&lt;njs`&gt; Yeah.</pre>
1186 </div>
1187 </div>
1188 <div class="paragraph">
1189 <p>Nuff said.</p>
1190 </div>
1191 <div class="literalblock">
1192 <div class="content">
1193 <pre>  &lt;linus&gt; Anyway.  I'm off for bed. It's not 6AM here, but I've got
1194 three kids, and have to get up early in the morning to send
1195 them off. I need my beauty sleep.</pre>
1196 </div>
1197 </div>
1198 <div class="literalblock">
1199 <div class="content">
1200 <pre>&lt;njs`&gt; :-)</pre>
1201 </div>
1202 </div>
1203 <div class="literalblock">
1204 <div class="content">
1205 <pre>   &lt;njs`&gt; appreciate the infodump, I really was failing to find the
1206 details on Git packs :-)</pre>
1207 </div>
1208 </div>
1209 <div class="paragraph">
1210 <p>And now you know the rest of the story.</p>
1211 </div>
1212 </div>
1213 <div id="footer">
1214 <div id="footer-text">
1215 Last updated 2020-03-10 15:02:33 -0700
1216 </div>
1217 </div>
1218 </body>
1219 </html>