From mboxrd@z Thu Jan 1 00:00:00 1970 From: Justin Luth To: development@lists.ipfire.org Subject: (attempt 2) Results of Updxlrator testing - working great Date: Fri, 12 Jan 2018 10:38:21 +0300 Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4557025799429510884==" List-Id: --===============4557025799429510884== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Updatexlrator testing - everything is working great!=C2=A0 (I compiled core=20 118, installed it on my spare ipfire replacement box, and swapped it in=20 yesterday morning.)=C2=A0 Here are some key excerpts from the debug log. The Microsoft files are generally defined as "URL" based because it is=20 not necessarily the name that is unique, but the url itself. 2018-01-11 08:26:58 [5878] [updxlrator] Running command=20 /var/ipfire/updatexlrator/bin/download microsoft=20 http://officecdn.microsoft.com/pr/39168D7E-077B-48E7-872C-B232C3E72675/Office= /Data/v32.cab=20 0 & 2018-01-11 08:27:04 [5878] [updxlrator] Running command=20 /var/ipfire/updatexlrator/bin/download microsoft=20 http://officecdn.microsoft.com/pr/492350f6-3a01-4f97-b9c0-c7c6ddf67d60/Office= /Data/v32.cab=20 0 & But if the filename contains the 40-digit SHA1 value, then we know that=20 the filename is unique, and so we can treat it like Linux files, where=20 it can be considered as a mirror-able file and only saved once. The=20 following logs highlight A.) Multiple requests for the same 2GB file.=20 (20 in two seconds!!) B.) The redirect from URL to FILE based caching.=C2=A0 = C.) two download requests before the lock file could be created.=20 (Imagine how many more would have started when the lock delayed until=20 wget wrote the first data.) 2018-01-11 17:35:23 [5878] [updxlrator] Processing=20 ...kb4056891-x64_5...=C2=A0 (first time this URL was seen.=C2=A0 22 attempts = in=20 the following 6 seconds). 2018-01-11 17:35:23 [6040] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:24 [6040] [updxlrator] SHA1: microsoft=20 033ed1d0-1709-957f-9fd1-34e24770fed4 not cached. Reprocessing as mirror=20 http://7.au.download.windowsupdate.com/c/msdownload/update/software/secu/2018= /01/windows10.0-kb4056891-x64_5_0a5255f85efca77e0abaa9c74f1581b5d3994c47.psf 2018-01-11 17:35:24 [5878] [updxlrator] SHA1: microsoft=20 033ed1d0-1709-957f-9fd1-34e24770fed4 not cached. Reprocessing as mirror=20 http://7.au.download.windowsupdate.com/c/msdownload/update/software/secu/2018= /01/windows10.0-kb4056891-x64_5_0a5255f85efca77e0abaa9c74f1581b5d3994c47.psf 2018-01-11 17:35:24 [6040] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:24 [5878] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:25 [6040] [updxlrator] Running command=20 /var/ipfire/updatexlrator/bin/download microsoft ...kb4056891-x64_5... 1 & 2018-01-11 17:35:25 [5878] [updxlrator] Running command=20 /var/ipfire/updatexlrator/bin/download microsoft ...kb4056891-x64_5... 1 & 2018-01-11 17:35:28 [6025] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:28 [6040] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:28 [6123] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:28 [6131] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:28 [6065] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:28 [6112] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:28 [2384] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:28 [2397] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:28 [2471] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:28 [2481] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:29 [2519] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:29 [2520] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:29 [2491] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:29 [2533] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:29 [16847] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:29 [2643] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:29 [16923] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:29 [16854] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:29 [16952] [updxlrator] Processing ...kb4056891-x64_5... 2018-01-11 17:35:29 [16990] [updxlrator] Processing ...kb4056891-x64_5... These logs show 3 different URLs that a single client=20 (55.55.55.40/DESKTOP-5NNUJJJ.local) requested this 2GB file from. 2018-01-12 02:21:12 [6115] [updxlrator] SHA1: microsoft=20 033ed1d0-1709-957f-9fd1-34e24770fed4 not cached. Reprocessing as mirror=20 http://7.au.download.windowsupdate.com/c/msdownload/update/software/secu/2018= /01/windows10.0-kb4056891-x64_5_0a5255f85efca77e0abaa9c74f1581b5d3994c47.psf 2018-01-12 04:13:44 [5878] [updxlrator] SHA1: microsoft=20 b6e85197-9fcb-ffd1-0388-fd8adad6a4b1 not cached. Reprocessing as mirror=20 http://000498-1.l.windowsupdate.com/llnhost_4.au.download.windowsupdate.com/c= /msdownload/update/software/secu/2018/01/windows10.0-kb4056891-x64_5_0a5255f8= 5efca77e0abaa9c74f1581b5d3994c47.psf 2018-01-12 04:14:11 [5878] [updxlrator] SHA1: microsoft=20 e28b880b-1350-d70a-fd16-be58c4ce1e45 not cached. Reprocessing as mirror=20 http://4.au.download.windowsupdate.com/c/msdownload/update/software/secu/2018= /01/windows10.0-kb4056891-x64_5_0a5255f85efca77e0abaa9c74f1581b5d3994c47.psf Oh, and some logs just to show the cache working in action - downloaded=20 from one mirror, and fulfilled in a request from another mirror ... 2018-01-11 16:43:58 [6025] [updxlrator] Running command=20 /var/ipfire/updatexlrator/bin/download microsoft=20 http://7.au.download.windowsupdate.com/c/msdownload/update/software/secu/2018= /01/windows10.0-kb4056891-x64_1_0a04efc071ff44c3262927fd2df414474e6a4c56.psf = 1 & ... 2018-01-12 03:22:03 [6025] [updxlrator] Processing URL=20 http://000f31-1.l.windowsupdate.com/llnhost_4.au.download.windowsupdate.com/c= /msdownload/update/software/secu/2018/01/windows10.0-kb4056891-x64_1_0a04efc0= 71ff44c3262927fd2df414474e6a4c56.psf 2018-01-12 03:22:03 [6025] [updxlrator] UUID is=20 da895428-94cf-268c-6545-56d4c1c1aca2 2018-01-12 03:22:04 [6025] [updxlrator] SHA1: microsoft=20 da895428-94cf-268c-6545-56d4c1c1aca2 not cached. Reprocessing as mirror=20 http://000f31-1.l.windowsupdate.com/llnhost_4.au.download.windowsupdate.com/c= /msdownload/update/software/secu/2018/01/windows10.0-kb4056891-x64_1_0a04efc0= 71ff44c3262927fd2df414474e6a4c56.psf 2018-01-12 03:22:04 [6025] [updxlrator] UUID is=20 b52663ab-8332-5292-0157-b95d50a5c96f 2018-01-12 03:22:04 [6040] [updxlrator] File exists in cache and is up=20 to date 2018-01-12 03:22:04 [6040] [updxlrator] Retrieving file from cache=20 (UPDCACHE) for 55.55.55.40/DESKTOP-5NNUJJJ.local So, yesterday's log nicely highlights the importance of these patches. -it was super easy (especially on slow connections) to have MANY=20 download attempts for the same file.=C2=A0 The download command now has=20 improved logic to prevent simultaneous wget's. I confirmed that only one=20 wget was running for each downloading file. -clients can request these huge updates from anywhere. Even though you=20 might have had the file cached from a previous request, if another=20 client starts requesting from a different mirror, you would have=20 downloaded that too.=C2=A0 The bandwidth (and disk) savings from this change = in logic is going to be huge. --===============4557025799429510884==--