Development February 2019

development@lists.ipfire.org

14 participants
55 discussions

[PATCH 00/20] Suricata Configuration Updates
by Michael Tremer 03 Mar '19

03 Mar '19

I have worked on suricata's configuration. My objective was to use more system resources (because suricata did not use much RAM, etc.) to make it faster and to be able to have some deeper decoding and matching. Please review these changes and let me know what you think. All in all, suricata should not use more than 1G of RAM which I think is a very good amount. If your system is weaker than that, there is no point in running an IPS. On my system in my office, this runs with a hand full of rules enabled from the Emerging Threats Community set at around 110MB of RAM. Michael Tremer (20): Revert "Suricata: detect DNS events on port 853, too" suricata: Set max-pending-packets to 1024 suricata: Set default packet size to 1514 suricata: Set detection profile to high suricata: Drop profiling section from configuration suricata: Drop some commented stuff from configuration suricata: Drop sections that require Rust suricata: Configure HTTP decoder suricata: Allow 32MB of RAM for DNS decoding suricata: Drop parsers I have never heard of suricata: We do not use any IP reputation lists suricata: Log to syslog suricata: Use the correct path for the magic database suricata: Use 64MB of RAM for defragmentation suricata: Use up to 256MB of RAM for the flow cache suricata: Log to syslog like a normal process suricata: Increase memory size for the stream engine suricata: Disable decoding for Teredo suricata: Start capture first and then load rules suricata: Fix syntax error config/etc/syslog.conf | 2 +- config/suricata/suricata.yaml | 282 +++++------------------------------------- 2 files changed, 30 insertions(+), 254 deletions(-) -- 2.12.2

2 26

[PATCH] Suricata: detect TLS traffic on port 444, too
by Peter Müller 01 Mar '19

01 Mar '19

This is the default port for IPFire's administrative web interface and should be monitored by Suricata, too. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> c: Stefan Schantl <stefan.schantl(a)ipfire.org> --- config/suricata/suricata.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/suricata/suricata.yaml b/config/suricata/suricata.yaml index 4fbd32b85..0ff06f4ae 100644 --- a/config/suricata/suricata.yaml +++ b/config/suricata/suricata.yaml @@ -140,7 +140,7 @@ app-layer: tls: enabled: yes detection-ports: - dp: "[443,465,993,995]" + dp: "[443,444,465,993,995]" # Completely stop processing TLS/SSL session after the handshake # completed. If bypass is enabled this will also trigger flow -- 2.16.4

3 2

[PATCH] suricata: Use highest bit to mark packets
by Michael Tremer 01 Mar '19

01 Mar '19

We are using the netfilter MARK in IPsec & QoS and this is causing conflicts. Therefore, we use the highest bit in the IPS chain now and clear it afterwards because we do not really care about this after the packets have been passed through suricata. Then, no other application has to worry about suricata. Fixes: #12010 Signed-off-by: Arne Fitzenreiter <arne.fitzenreiter(a)ipfire.org> Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> --- config/suricata/suricata.yaml | 4 ++-- src/initscripts/system/suricata | 7 +++++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/config/suricata/suricata.yaml b/config/suricata/suricata.yaml index 12937ab22..7f651327e 100644 --- a/config/suricata/suricata.yaml +++ b/config/suricata/suricata.yaml @@ -117,8 +117,8 @@ logging: nfq: mode: repeat - repeat-mark: 16 - repeat-mask: 16 + repeat-mark: 1879048192 + repeat-mask: 1879048192 # bypass-mark: 1 # bypass-mask: 1 # route-queue: 2 diff --git a/src/initscripts/system/suricata b/src/initscripts/system/suricata index d2c758660..e755dfaff 100644 --- a/src/initscripts/system/suricata +++ b/src/initscripts/system/suricata @@ -29,8 +29,8 @@ NFQ_OPTS="--queue-bypass " network_zones=( red green blue orange ) # Mark and Mask options. -MARK="0x16" -MASK="0x16" +MARK="0x70000000" +MASK="0x70000000" # PID file of suricata. PID_FILE="/var/run/suricata.pid" @@ -88,6 +88,9 @@ function generate_fw_rules { iptables -I "$FW_CHAIN" -o "$network_device" -m mark ! --mark "$MARK"/"$MASK" -j NFQUEUE $NFQ_OPTIONS fi done + + # Clear repeat bit, so that it does not confuse IPsec or QoS + iptables -A "${FW_CHAIN}" -j MARK --set-xmark "0x0/${MASK}" } # Function to flush the firewall chain. -- 2.12.2

2 1

(x86_64) - core128 Orange network
by Mentalic 28 Feb '19

28 Feb '19

So I'm testing Orange network with Device that has a bad habit of broadcasting to 255.255.255.255 every minute or so. Thing is I don't want to see this logged as DROP_INPUT from Orange interface cluttering up the Firewall log. When I had this device on Blue I made a rule to capture this traffic then not log the rule. Had no success in creating a rule in Orange to capture this traffic before its logged in Firewall log. Perhaps the order of rule application is different on Orange? Regards Wayne

1 1

[PATCH 1/2] netsnmpd: Update to version 5.8
by Erik Kapfer 27 Feb '19

27 Feb '19

Overview of the changes can be found in here https://sourceforge.net/p/net-snmp/mailman/message/36386084/ . Signed-off-by: Erik Kapfer <ummeegge(a)ipfire.org> --- config/rootfiles/packages/netsnmpd | 163 +++++++++++++++++++++---------------- lfs/netsnmpd | 11 +-- 2 files changed, 100 insertions(+), 74 deletions(-) diff --git a/config/rootfiles/packages/netsnmpd b/config/rootfiles/packages/netsnmpd index 9d80ec2ad..39ae42056 100644 --- a/config/rootfiles/packages/netsnmpd +++ b/config/rootfiles/packages/netsnmpd @@ -1,8 +1,10 @@ +etc/rc.d/init.d/netsnmpd etc/rc.d/rc0.d/K02netsnmpd etc/rc.d/rc3.d/S65netsnmpd etc/rc.d/rc6.d/K02netsnmpd etc/snmpd.conf usr/bin/agentxtrap +usr/bin/checkbandwidth usr/bin/encode_keychange usr/bin/fixproc usr/bin/ipf-mod.pl @@ -22,10 +24,14 @@ usr/bin/snmpget usr/bin/snmpgetnext usr/bin/snmpinform usr/bin/snmpnetstat +usr/bin/snmppcap +usr/bin/snmpping +usr/bin/snmpps usr/bin/snmpset usr/bin/snmpstatus usr/bin/snmptable usr/bin/snmptest +usr/bin/snmptop usr/bin/snmptranslate usr/bin/snmptrap usr/bin/snmpusm @@ -58,6 +64,7 @@ usr/bin/traptoemail #usr/include/net-snmp/agent/mode_end_call.h #usr/include/net-snmp/agent/multiplexer.h #usr/include/net-snmp/agent/net-snmp-agent-includes.h +#usr/include/net-snmp/agent/netsnmp_close_fds.h #usr/include/net-snmp/agent/null.h #usr/include/net-snmp/agent/old_api.h #usr/include/net-snmp/agent/read_only.h @@ -114,6 +121,7 @@ usr/bin/traptoemail #usr/include/net-snmp/library/md5.h #usr/include/net-snmp/library/mib.h #usr/include/net-snmp/library/mt_support.h +#usr/include/net-snmp/library/netsnmp-attribute-format.h #usr/include/net-snmp/library/oid.h #usr/include/net-snmp/library/oid_stash.h #usr/include/net-snmp/library/parse.h @@ -124,12 +132,15 @@ usr/bin/traptoemail #usr/include/net-snmp/library/snmpAliasDomain.h #usr/include/net-snmp/library/snmpCallbackDomain.h #usr/include/net-snmp/library/snmpIPv4BaseDomain.h +#usr/include/net-snmp/library/snmpIPv6BaseDomain.h #usr/include/net-snmp/library/snmpSocketBaseDomain.h #usr/include/net-snmp/library/snmpTCPBaseDomain.h #usr/include/net-snmp/library/snmpTCPDomain.h +#usr/include/net-snmp/library/snmpTCPIPv6Domain.h #usr/include/net-snmp/library/snmpUDPBaseDomain.h #usr/include/net-snmp/library/snmpUDPDomain.h #usr/include/net-snmp/library/snmpUDPIPv4BaseDomain.h +#usr/include/net-snmp/library/snmpUDPIPv6Domain.h #usr/include/net-snmp/library/snmpUnixDomain.h #usr/include/net-snmp/library/snmp_alarm.h #usr/include/net-snmp/library/snmp_api.h @@ -174,6 +185,13 @@ usr/bin/traptoemail #usr/include/net-snmp/system/cygwin.h #usr/include/net-snmp/system/darwin.h #usr/include/net-snmp/system/darwin10.h +#usr/include/net-snmp/system/darwin11.h +#usr/include/net-snmp/system/darwin12.h +#usr/include/net-snmp/system/darwin13.h +#usr/include/net-snmp/system/darwin14.h +#usr/include/net-snmp/system/darwin15.h +#usr/include/net-snmp/system/darwin16.h +#usr/include/net-snmp/system/darwin17.h #usr/include/net-snmp/system/darwin7.h #usr/include/net-snmp/system/darwin8.h #usr/include/net-snmp/system/darwin9.h @@ -194,13 +212,17 @@ usr/bin/traptoemail #usr/include/net-snmp/system/generic.h #usr/include/net-snmp/system/hpux.h #usr/include/net-snmp/system/irix.h +#usr/include/net-snmp/system/kfreebsd.h #usr/include/net-snmp/system/linux.h #usr/include/net-snmp/system/mingw32.h +#usr/include/net-snmp/system/mingw32msvc.h #usr/include/net-snmp/system/mips.h #usr/include/net-snmp/system/netbsd.h +#usr/include/net-snmp/system/nto-qnx6.h #usr/include/net-snmp/system/openbsd.h #usr/include/net-snmp/system/openbsd4.h #usr/include/net-snmp/system/openbsd5.h +#usr/include/net-snmp/system/openbsd6.h #usr/include/net-snmp/system/osf5.h #usr/include/net-snmp/system/solaris.h #usr/include/net-snmp/system/solaris2.3.h @@ -217,74 +239,74 @@ usr/bin/traptoemail #usr/include/net-snmp/version.h #usr/lib/libnetsnmp.a #usr/lib/libnetsnmp.la -usr/lib/libnetsnmp.so -usr/lib/libnetsnmp.so.30 -usr/lib/libnetsnmp.so.30.0.3 +#usr/lib/libnetsnmp.so +usr/lib/libnetsnmp.so.35 +usr/lib/libnetsnmp.so.35.0.0 #usr/lib/libnetsnmpagent.a #usr/lib/libnetsnmpagent.la -usr/lib/libnetsnmpagent.so -usr/lib/libnetsnmpagent.so.30 -usr/lib/libnetsnmpagent.so.30.0.3 +#usr/lib/libnetsnmpagent.so +usr/lib/libnetsnmpagent.so.35 +usr/lib/libnetsnmpagent.so.35.0.0 #usr/lib/libnetsnmphelpers.a #usr/lib/libnetsnmphelpers.la -usr/lib/libnetsnmphelpers.so -usr/lib/libnetsnmphelpers.so.30 -usr/lib/libnetsnmphelpers.so.30.0.3 +#usr/lib/libnetsnmphelpers.so +usr/lib/libnetsnmphelpers.so.35 +usr/lib/libnetsnmphelpers.so.35.0.0 #usr/lib/libnetsnmpmibs.a #usr/lib/libnetsnmpmibs.la -usr/lib/libnetsnmpmibs.so -usr/lib/libnetsnmpmibs.so.30 -usr/lib/libnetsnmpmibs.so.30.0.3 +#usr/lib/libnetsnmpmibs.so +usr/lib/libnetsnmpmibs.so.35 +usr/lib/libnetsnmpmibs.so.35.0.0 #usr/lib/libnetsnmptrapd.a #usr/lib/libnetsnmptrapd.la -usr/lib/libnetsnmptrapd.so -usr/lib/libnetsnmptrapd.so.30 -usr/lib/libnetsnmptrapd.so.30.0.3 -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/Bundle -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/Bundle/Makefile.subs.pl -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/NetSNMP -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/NetSNMP/ASN.pm -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/NetSNMP/OID.pm -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/NetSNMP/TrapReceiver.pm -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/NetSNMP/agent -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/NetSNMP/agent.pm -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/NetSNMP/agent/Support.pm -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/NetSNMP/agent/default_store.pm -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/NetSNMP/agent/netsnmp_request_infoPtr.pm -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/NetSNMP/default_store.pm -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/SNMP.pm -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/Bundle -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/Bundle/NetSNMP -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/Bundle/NetSNMP/.packlist -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/ASN -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/ASN/ASN.bs -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/ASN/ASN.so -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/ASN/autosplit.ix -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/OID -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/OID/OID.bs -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/OID/OID.so -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/OID/autosplit.ix -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/TrapReceiver -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/TrapReceiver/TrapReceiver.bs -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/TrapReceiver/TrapReceiver.so -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/TrapReceiver/autosplit.ix -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/agent -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/agent/agent.bs -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/agent/agent.so -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/agent/autosplit.ix -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/agent/default_store -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/agent/default_store/autosplit.ix -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/agent/default_store/default_store.bs -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/agent/default_store/default_store.so -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/default_store -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/default_store/autosplit.ix -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/default_store/default_store.bs -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/NetSNMP/default_store/default_store.so -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/SNMP -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/SNMP/SNMP.bs -usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/SNMP/SNMP.so -#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/SNMP/autosplit.ix +#usr/lib/libnetsnmptrapd.so +usr/lib/libnetsnmptrapd.so.35 +usr/lib/libnetsnmptrapd.so.35.0.0 +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/Bundle +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/Bundle/MakefileSubs.pm +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/NetSNMP +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/NetSNMP/ASN.pm +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/NetSNMP/OID.pm +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/NetSNMP/TrapReceiver.pm +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/NetSNMP/agent +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/NetSNMP/agent.pm +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/NetSNMP/agent/Support.pm +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/NetSNMP/agent/default_store.pm +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/NetSNMP/agent/netsnmp_request_infoPtr.pm +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/NetSNMP/default_store.pm +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/SNMP.pm +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/Bundle +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/Bundle/NetSNMP +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/Bundle/NetSNMP/.packlist +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/ASN +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/ASN/ASN.bs +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/ASN/ASN.so +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/ASN/autosplit.ix +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/OID +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/OID/OID.bs +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/OID/OID.so +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/OID/autosplit.ix +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/TrapReceiver +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/TrapReceiver/TrapReceiver.bs +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/TrapReceiver/TrapReceiver.so +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/TrapReceiver/autosplit.ix +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/agent +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/agent/agent.bs +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/agent/agent.so +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/agent/autosplit.ix +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/agent/default_store +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/agent/default_store/autosplit.ix +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/agent/default_store/default_store.bs +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/agent/default_store/default_store.so +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/default_store +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/default_store/autosplit.ix +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/default_store/default_store.bs +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/NetSNMP/default_store/default_store.so +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/SNMP +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/SNMP/SNMP.bs +usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/SNMP/SNMP.so +#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/SNMP/autosplit.ix usr/sbin/snmpd usr/sbin/snmptrapd #usr/share/man/man1/agentxtrap.1 @@ -305,10 +327,12 @@ usr/sbin/snmptrapd #usr/share/man/man1/snmpgetnext.1 #usr/share/man/man1/snmpinform.1 #usr/share/man/man1/snmpnetstat.1 +#usr/share/man/man1/snmpps.1 #usr/share/man/man1/snmpset.1 #usr/share/man/man1/snmpstatus.1 #usr/share/man/man1/snmptable.1 #usr/share/man/man1/snmptest.1 +#usr/share/man/man1/snmptop.1 #usr/share/man/man1/snmptranslate.1 #usr/share/man/man1/snmptrap.1 #usr/share/man/man1/snmpusm.1 @@ -427,8 +451,8 @@ usr/sbin/snmptrapd #usr/share/man/man5/variables.5 #usr/share/man/man8/snmpd.8 #usr/share/man/man8/snmptrapd.8 -usr/share/snmp -usr/share/snmp/mib2c-data +#usr/share/snmp +#usr/share/snmp/mib2c-data usr/share/snmp/mib2c-data/default-mfd-top.m2c usr/share/snmp/mib2c-data/details-enums.m2i usr/share/snmp/mib2c-data/details-node.m2i @@ -513,11 +537,12 @@ usr/share/snmp/mib2c.iterate_access.conf usr/share/snmp/mib2c.mfd.conf usr/share/snmp/mib2c.notify.conf usr/share/snmp/mib2c.old-api.conf +usr/share/snmp/mib2c.org-mode.conf usr/share/snmp/mib2c.perl.conf usr/share/snmp/mib2c.raw-table.conf usr/share/snmp/mib2c.scalar.conf usr/share/snmp/mib2c.table_data.conf -usr/share/snmp/mibs +#usr/share/snmp/mibs usr/share/snmp/mibs/AGENTX-MIB.txt usr/share/snmp/mibs/BRIDGE-MIB.txt usr/share/snmp/mibs/DISMAN-EVENT-MIB.txt @@ -570,6 +595,7 @@ usr/share/snmp/mibs/SNMP-TSM-MIB.txt usr/share/snmp/mibs/SNMP-USER-BASED-SM-MIB.txt usr/share/snmp/mibs/SNMP-USM-AES-MIB.txt usr/share/snmp/mibs/SNMP-USM-DH-OBJECTS-MIB.txt +usr/share/snmp/mibs/SNMP-USM-HMAC-SHA2-MIB.txt usr/share/snmp/mibs/SNMP-VIEW-BASED-ACM-MIB.txt usr/share/snmp/mibs/SNMPv2-CONF.txt usr/share/snmp/mibs/SNMPv2-MIB.txt @@ -587,14 +613,14 @@ usr/share/snmp/mibs/UCD-SNMP-MIB.txt usr/share/snmp/mibs/UDP-MIB.txt usr/share/snmp/snmp_perl.pl usr/share/snmp/snmp_perl_trapd.pl -usr/share/snmp/snmpconf-data -usr/share/snmp/snmpconf-data/snmp-data +#usr/share/snmp/snmpconf-data +#usr/share/snmp/snmpconf-data/snmp-data usr/share/snmp/snmpconf-data/snmp-data/authopts usr/share/snmp/snmpconf-data/snmp-data/debugging usr/share/snmp/snmpconf-data/snmp-data/mibs usr/share/snmp/snmpconf-data/snmp-data/output usr/share/snmp/snmpconf-data/snmp-data/snmpconf-config -usr/share/snmp/snmpconf-data/snmpd-data +#usr/share/snmp/snmpconf-data/snmpd-data usr/share/snmp/snmpconf-data/snmpd-data/acl usr/share/snmp/snmpconf-data/snmpd-data/basic_setup usr/share/snmp/snmpconf-data/snmpd-data/extending @@ -603,12 +629,11 @@ usr/share/snmp/snmpconf-data/snmpd-data/operation usr/share/snmp/snmpconf-data/snmpd-data/snmpconf-config usr/share/snmp/snmpconf-data/snmpd-data/system usr/share/snmp/snmpconf-data/snmpd-data/trapsinks -usr/share/snmp/snmpconf-data/snmptrapd-data +#usr/share/snmp/snmpconf-data/snmptrapd-data usr/share/snmp/snmpconf-data/snmptrapd-data/authentication usr/share/snmp/snmpconf-data/snmptrapd-data/formatting usr/share/snmp/snmpconf-data/snmptrapd-data/logging usr/share/snmp/snmpconf-data/snmptrapd-data/runtime usr/share/snmp/snmpconf-data/snmptrapd-data/snmpconf-config usr/share/snmp/snmpconf-data/snmptrapd-data/traphandle -var/ipfire/backup/addons/includes/netsnmpd -etc/rc.d/init.d/netsnmpd +var/ipfire/backup/addons/includes/netsnmpd \ No newline at end of file diff --git a/lfs/netsnmpd b/lfs/netsnmpd index 06233f3e9..0af276093 100644 --- a/lfs/netsnmpd +++ b/lfs/netsnmpd @@ -1,7 +1,7 @@ ############################################################################### # # # IPFire.org - A linux based firewall # -# Copyright (C) 2007-2018 IPFire Team <info(a)ipfire.org> # +# Copyright (C) 2007-2019 IPFire Team <info(a)ipfire.org> # # # # This program is free software: you can redistribute it and/or modify # # it under the terms of the GNU General Public License as published by # @@ -24,7 +24,7 @@ include Config -VER = 5.7.3 +VER = 5.8 THISAPP = net-snmp-$(VER) DL_FILE = $(THISAPP).tar.gz @@ -32,7 +32,7 @@ DL_FROM = $(URL_IPFIRE) DIR_APP = $(DIR_SRC)/$(THISAPP) TARGET = $(DIR_INFO)/$(THISAPP) PROG = netsnmpd -PAK_VER = 7 +PAK_VER = 8 DEPS = "" @@ -44,7 +44,7 @@ objects = $(DL_FILE) $(DL_FILE) = $(DL_FROM)/$(DL_FILE) -$(DL_FILE)_MD5 = d4a3459e1577d0efa8d96ca70a885e53 +$(DL_FILE)_MD5 = 63bfc65fbb86cdb616598df1aff6458a install : $(TARGET) @@ -77,7 +77,7 @@ $(subst %,%_MD5,$(objects)) : $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @$(PREBUILD) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE) - cd $(DIR_APP) && patch -Np1 < $(DIR_SRC)/src/patches/net-snmp-5.7.3-openssl.patch + $(UPDATE_AUTOMAKE) cd $(DIR_APP) && ./configure \ --prefix=/usr \ @@ -95,6 +95,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) sctp-mib rmon-mib etherlike-mib ucd-snmp/lmsensorsMib" --libdir=/usr/lib \ --sysconfdir="/etc" + cd $(DIR_APP) && make cd $(DIR_APP) && make install install -v -m 644 $(DIR_SRC)/config/netsnmpd/snmpd.conf /etc/snmpd.conf -- 2.12.2

1 1

[PATCH] OpenVPN: Update to version 2.4.7
by Erik Kapfer 26 Feb '19

26 Feb '19

Changelog can be found in here https://community.openvpn.net/openvpn/wiki/ChangesInOpenvpn24 . Signed-off-by: Erik Kapfer <ummeegge(a)ipfire.org> --- lfs/openvpn | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lfs/openvpn b/lfs/openvpn index 2503654f1..61c805fdb 100644 --- a/lfs/openvpn +++ b/lfs/openvpn @@ -1,7 +1,7 @@ ############################################################################### # # # IPFire.org - A linux based firewall # -# Copyright (C) 2007-2018 IPFire Team <info(a)ipfire.org> # +# Copyright (C) 2007-2019 IPFire Team <info(a)ipfire.org> # # # # This program is free software: you can redistribute it and/or modify # # it under the terms of the GNU General Public License as published by # @@ -24,7 +24,7 @@ include Config -VER = 2.4.6 +VER = 2.4.7 THISAPP = openvpn-$(VER) DL_FILE = $(THISAPP).tar.xz @@ -40,7 +40,7 @@ objects = $(DL_FILE) $(DL_FILE) = $(DL_FROM)/$(DL_FILE) -$(DL_FILE)_MD5 = 3a1f3f63bdaede443b4df49957df9405 +$(DL_FILE)_MD5 = 4ad8a008e1e7f261b3aa0024e79e7fb7 install : $(TARGET) -- 2.12.2

3 2

[PATCH 1/2] update metrics links in Tor WebUI
by Peter Müller 26 Feb '19

26 Feb '19

https://atlas.torproject.org/ is deprecated in favour of https://metrics.torproject.org/ by now. Fixes #11781. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- html/cgi-bin/tor.cgi | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/html/cgi-bin/tor.cgi b/html/cgi-bin/tor.cgi index 0d235c949..b17db01a2 100644 --- a/html/cgi-bin/tor.cgi +++ b/html/cgi-bin/tor.cgi @@ -519,7 +519,7 @@ END <tr> <td width='40%' class='base'>$Lang::tr{'tor relay fingerprint'}:</td> <td width='60%'> - <a href='https://atlas.torproject.org/#details/$fingerprint' target='_blank'>$fingerprint</a> + <a href='https://metrics.torproject.org/rs.html#details/$fingerprint' target='_blank'>$fingerprint</a> </td> </tr> END @@ -612,7 +612,7 @@ END print <<END; <tr> <td width='40%'> - <a href='https://atlas.torproject.org/#details/$node->{'fingerprint'}' target='_blank'> + <a href='https://metrics.torproject.org/rs.html#details/$node->{'fingerprint'}' target='_blank'> $node->{'name'} </a> </td> -- 2.16.4

2 1

[PATCH] libgcrypt: update to 1.8.4
by Peter Müller 26 Feb '19

26 Feb '19

Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- config/rootfiles/common/libgcrypt | 2 +- lfs/libgcrypt | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/config/rootfiles/common/libgcrypt b/config/rootfiles/common/libgcrypt index e092ebbc7..efd9ac46a 100644 --- a/config/rootfiles/common/libgcrypt +++ b/config/rootfiles/common/libgcrypt @@ -6,7 +6,7 @@ #usr/lib/libgcrypt.la #usr/lib/libgcrypt.so usr/lib/libgcrypt.so.20 -usr/lib/libgcrypt.so.20.2.3 +usr/lib/libgcrypt.so.20.2.4 #usr/share/aclocal/libgcrypt.m4 #usr/share/info/gcrypt.info #usr/share/info/gcrypt.info-1 diff --git a/lfs/libgcrypt b/lfs/libgcrypt index ec99d936b..5beefbf12 100644 --- a/lfs/libgcrypt +++ b/lfs/libgcrypt @@ -1,7 +1,7 @@ ############################################################################### # # # IPFire.org - A linux based firewall # -# Copyright (C) 2007-2018 IPFire Team <info(a)ipfire.org> # +# Copyright (C) 2007-2019 IPFire Team <info(a)ipfire.org> # # # # This program is free software: you can redistribute it and/or modify # # it under the terms of the GNU General Public License as published by # @@ -24,7 +24,7 @@ include Config -VER = 1.8.3 +VER = 1.8.4 THISAPP = libgcrypt-$(VER) DL_FILE = $(THISAPP).tar.bz2 @@ -40,7 +40,7 @@ objects = $(DL_FILE) $(DL_FILE) = $(DL_FROM)/$(DL_FILE) -$(DL_FILE)_MD5 = 3139c2402e844985a67fb288a930534d +$(DL_FILE)_MD5 = fbfdaebbbc6d7e5fbbf6ffdb3e139573 install : $(TARGET) -- 2.16.4

2 1

[PATCH] unbound: Update to 1.9.0
by Matthias Fischer 26 Feb '19

26 Feb '19

For details see: https://nlnetlabs.nl/svn/unbound/tags/release-1.9.0/doc/Changelog Best, Matthias Signed-off-by: Matthias Fischer <matthias.fischer(a)ipfire.org> --- config/rootfiles/common/unbound | 2 +- lfs/unbound | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/config/rootfiles/common/unbound b/config/rootfiles/common/unbound index 9a8126c15..843e0eeca 100644 --- a/config/rootfiles/common/unbound +++ b/config/rootfiles/common/unbound @@ -11,7 +11,7 @@ etc/unbound/unbound.conf #usr/lib/libunbound.la #usr/lib/libunbound.so usr/lib/libunbound.so.8 -usr/lib/libunbound.so.8.0.3 +usr/lib/libunbound.so.8.1.0 #usr/lib/pkgconfig/libunbound.pc usr/sbin/unbound usr/sbin/unbound-anchor diff --git a/lfs/unbound b/lfs/unbound index 07501d1d6..b090010d4 100644 --- a/lfs/unbound +++ b/lfs/unbound @@ -24,7 +24,7 @@ include Config -VER = 1.8.3 +VER = 1.9.0 THISAPP = unbound-$(VER) DL_FILE = $(THISAPP).tar.gz @@ -40,7 +40,7 @@ objects = $(DL_FILE) $(DL_FILE) = $(DL_FROM)/$(DL_FILE) -$(DL_FILE)_MD5 = 4646203343d3b8f5aeb1b57753c27ead +$(DL_FILE)_MD5 = 1026159991a3883518525bc18e25582f install : $(TARGET) -- 2.18.0

3 8

[PATCH] squid 4.5: latest patches (01-09)
by Matthias Fischer 25 Feb '19

25 Feb '19

For details see: http://www.squid-cache.org/Versions/v4/changesets/ Best, Matthias Signed-off-by: Matthias Fischer <matthias.fischer(a)ipfire.org> --- lfs/squid | 11 +- ...ile_errors_with_-O3_optimization_288.patch | 107 ++++++++ ...GoIntoBackground_fork_call_fails_344.patch | 29 +++ ...associated_with_auto-consumption_348.patch | 118 +++++++++ ...ds_that_define_OPENSSL_NO_ENGINE_349.patch | 22 ++ ...disker-to-worker_queue_overflows_353.patch | 92 +++++++ ..._when_reserving_a_new_cache_slot_350.patch | 246 ++++++++++++++++++ ...opped_some_of_the_write_requests_352.patch | 164 ++++++++++++ ...call_setsid_in_--foreground_mode_354.patch | 36 +++ ...ect_IPv6_loopback_binding_errors_355.patch | 54 ++++ 10 files changed, 878 insertions(+), 1 deletion(-) create mode 100644 src/patches/squid/01_Bug_4875_pt2_GCC-8_compile_errors_with_-O3_optimization_288.patch create mode 100644 src/patches/squid/02_Bug_4856_Exit_when_GoIntoBackground_fork_call_fails_344.patch create mode 100644 src/patches/squid/03_Fix_BodyPipe_Sink_memory_leaks_associated_with_auto-consumption_348.patch create mode 100644 src/patches/squid/04_Fix_OpenSSL_builds_that_define_OPENSSL_NO_ENGINE_349.patch create mode 100644 src/patches/squid/05_Fixed_disker-to-worker_queue_overflows_353.patch create mode 100644 src/patches/squid/06_Initialize_StoreMapSlice_when_reserving_a_new_cache_slot_350.patch create mode 100644 src/patches/squid/07_Fail_Rock_swapout_if_the_disk_dropped_some_of_the_write_requests_352.patch create mode 100644 src/patches/squid/08_Bug_4914_Do_not_call_setsid_in_--foreground_mode_354.patch create mode 100644 src/patches/squid/09_Bug_4915_Detect_IPv6_loopback_binding_errors_355.patch diff --git a/lfs/squid b/lfs/squid index 6033ab394..6c4706240 100644 --- a/lfs/squid +++ b/lfs/squid @@ -72,6 +72,15 @@ $(subst %,%_MD5,$(objects)) : $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @$(PREBUILD) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar xaf $(DIR_DL)/$(DL_FILE) + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/squid/01_Bug_4875_pt2_GCC-8_compile_errors_with_-O3_optimization_288.patch + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/squid/02_Bug_4856_Exit_when_GoIntoBackground_fork_call_fails_344.patch + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/squid/03_Fix_BodyPipe_Sink_memory_leaks_associated_with_auto-consumption_348.patch + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/squid/04_Fix_OpenSSL_builds_that_define_OPENSSL_NO_ENGINE_349.patch + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/squid/05_Fixed_disker-to-worker_queue_overflows_353.patch + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/squid/06_Initialize_StoreMapSlice_when_reserving_a_new_cache_slot_350.patch + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/squid/07_Fail_Rock_swapout_if_the_disk_dropped_some_of_the_write_requests_352.patch + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/squid/08_Bug_4914_Do_not_call_setsid_in_--foreground_mode_354.patch + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/squid/09_Bug_4915_Detect_IPv6_loopback_binding_errors_355.patch cd $(DIR_APP) && patch -Np0 -i $(DIR_SRC)/src/patches/squid/squid-4.5-fix-max-file-descriptors.patch cd $(DIR_APP) && autoreconf -vfi @@ -91,7 +100,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) --disable-kqueue \ --disable-esi \ --disable-arch-native \ - --enable-ipv6 \ + --disable-ipv6 \ --enable-poll \ --enable-ident-lookups \ --enable-storeio=aufs,diskd,ufs \ diff --git a/src/patches/squid/01_Bug_4875_pt2_GCC-8_compile_errors_with_-O3_optimization_288.patch b/src/patches/squid/01_Bug_4875_pt2_GCC-8_compile_errors_with_-O3_optimization_288.patch new file mode 100644 index 000000000..1e3b731a0 --- /dev/null +++ b/src/patches/squid/01_Bug_4875_pt2_GCC-8_compile_errors_with_-O3_optimization_288.patch @@ -0,0 +1,107 @@ +commit d6adda4644e614dcaa6831abc5efffb2af51a7c8 +Author: Amos Jeffries <yadij(a)users.noreply.github.com> +Date: 2019-01-06 13:22:19 +0000 + + Bug 4875 pt2: GCC-8 compile errors with -O3 optimization (#288) + + GCC-8 warnings exposed at -O3 optimization causes its + own static analyzer to detect optimized code is eliding + initialization on paths that do not use the + configuration variables. + + Refactor the parseTimeLine() API to return the parsed + values so that there is no need to initialize anything prior + to parsing. + +diff --git a/src/cache_cf.cc b/src/cache_cf.cc +index c4eb955..a45ea2e 100644 +--- a/src/cache_cf.cc ++++ b/src/cache_cf.cc +@@ -160,7 +160,6 @@ static bool setLogformat(CustomLog *cl, const char *name, const bool dieWhenMiss + static void configDoConfigure(void); + static void parse_refreshpattern(RefreshPattern **); + static uint64_t parseTimeUnits(const char *unit, bool allowMsec); +-static void parseTimeLine(time_msec_t * tptr, const char *units, bool allowMsec, bool expectMoreArguments); + static void parse_u_short(unsigned short * var); + static void parse_string(char **); + static void default_all(void); +@@ -1034,19 +1033,19 @@ parse_obsolete(const char *name) + + /* Parse a time specification from the config file. Store the + * result in 'tptr', after converting it to 'units' */ +-static void +-parseTimeLine(time_msec_t * tptr, const char *units, bool allowMsec, bool expectMoreArguments = false) ++static time_msec_t ++parseTimeLine(const char *units, bool allowMsec, bool expectMoreArguments = false) + { + time_msec_t u = parseTimeUnits(units, allowMsec); + if (u == 0) { + self_destruct(); +- return; ++ return 0; + } + +- char *token = ConfigParser::NextToken();; ++ char *token = ConfigParser::NextToken(); + if (!token) { + self_destruct(); +- return; ++ return 0; + } + + double d = xatof(token); +@@ -1059,7 +1058,7 @@ parseTimeLine(time_msec_t * tptr, const char *units, bool allowMsec, bool expe + + } else if (!expectMoreArguments) { + self_destruct(); +- return; ++ return 0; + + } else { + token = NULL; // show default units if dying below +@@ -1068,13 +1067,14 @@ parseTimeLine(time_msec_t * tptr, const char *units, bool allowMsec, bool expe + } else + token = NULL; // show default units if dying below. + +- *tptr = static_cast<time_msec_t>(m * d); ++ const auto result = static_cast<time_msec_t>(m * d); + +- if (static_cast<double>(*tptr) * 2 != m * d * 2) { ++ if (static_cast<double>(result) * 2 != m * d * 2) { + debugs(3, DBG_CRITICAL, "FATAL: Invalid value '" << + d << " " << (token ? token : units) << ": integer overflow (time_msec_t)."); + self_destruct(); + } ++ return result; + } + + static uint64_t +@@ -2918,8 +2918,7 @@ dump_time_t(StoreEntry * entry, const char *name, time_t var) + void + parse_time_t(time_t * var) + { +- time_msec_t tval; +- parseTimeLine(&tval, T_SECOND_STR, false); ++ time_msec_t tval = parseTimeLine(T_SECOND_STR, false); + *var = static_cast<time_t>(tval/1000); + } + +@@ -2941,7 +2940,7 @@ dump_time_msec(StoreEntry * entry, const char *name, time_msec_t var) + void + parse_time_msec(time_msec_t * var) + { +- parseTimeLine(var, T_SECOND_STR, true); ++ *var = parseTimeLine(T_SECOND_STR, true); + } + + static void +@@ -4814,8 +4813,7 @@ static void free_ftp_epsv(acl_access **ftp_epsv) + static void + parse_UrlHelperTimeout(SquidConfig::UrlHelperTimeout *config) + { +- time_msec_t tval; +- parseTimeLine(&tval, T_SECOND_STR, false, true); ++ const auto tval = parseTimeLine(T_SECOND_STR, false, true); + Config.Timeout.urlRewrite = static_cast<time_t>(tval/1000); + + char *key, *value; diff --git a/src/patches/squid/02_Bug_4856_Exit_when_GoIntoBackground_fork_call_fails_344.patch b/src/patches/squid/02_Bug_4856_Exit_when_GoIntoBackground_fork_call_fails_344.patch new file mode 100644 index 000000000..3159b9d1c --- /dev/null +++ b/src/patches/squid/02_Bug_4856_Exit_when_GoIntoBackground_fork_call_fails_344.patch @@ -0,0 +1,29 @@ +commit 6aa83b8d51dbb66bf5383f18615a59d8ff0a10a4 +Author: Marcos Mello <marcosfrm(a)users.noreply.github.com> +Date: 2019-01-08 10:14:09 +0000 + + Bug 4856: Exit when GoIntoBackground() fork() call fails (#344) + + Ignoring the (implicit/default) command-line instruction to background + Squid is wrong in general and confuses at least some daemon managers. + +diff --git a/src/main.cc b/src/main.cc +index eeb8f72..7f5ec80 100644 +--- a/src/main.cc ++++ b/src/main.cc +@@ -1865,13 +1865,12 @@ GoIntoBackground() + pid_t pid; + if ((pid = fork()) < 0) { + int xerrno = errno; +- syslog(LOG_ALERT, "fork failed: %s", xstrerr(xerrno)); +- // continue anyway, mimicking --foreground mode (XXX?) ++ throw TexcHere(ToSBuf("failed to fork(2) the master process: ", xstrerr(xerrno))); + } else if (pid > 0) { + // parent + exit(EXIT_SUCCESS); + } +- // child, running as a background daemon (or a failed-to-fork parent) ++ // child, running as a background daemon + } + + static void diff --git a/src/patches/squid/03_Fix_BodyPipe_Sink_memory_leaks_associated_with_auto-consumption_348.patch b/src/patches/squid/03_Fix_BodyPipe_Sink_memory_leaks_associated_with_auto-consumption_348.patch new file mode 100644 index 000000000..efb0d4c69 --- /dev/null +++ b/src/patches/squid/03_Fix_BodyPipe_Sink_memory_leaks_associated_with_auto-consumption_348.patch @@ -0,0 +1,118 @@ +commit dd9d539a35275071d7718cfbdbc3203d7aa05139 +Author: Alex Rousskov <rousskov(a)measurement-factory.com> +Date: 2019-01-08 15:14:18 +0000 + + Fix BodyPipe/Sink memory leaks associated with auto-consumption (#348) + + Auto-consumption happens (and could probably leak memory) in many cases, + but this leak was exposed by an eCAP service that blocked or replaced + virgin messages. + + The BodySink job termination algorithm relies on body production + notifications. A BodySink job created after the body production had + ended can never stop and, hence, leaks (leaking the associated BodyPipe + object with it). Such a job is also useless: If production is over, + there is no need to free space for more body data! This change avoids + creating such leaking and useless jobs. + +diff --git a/src/BodyPipe.cc b/src/BodyPipe.cc +index 0fdf0dc..59477bd 100644 +--- a/src/BodyPipe.cc ++++ b/src/BodyPipe.cc +@@ -286,8 +286,7 @@ BodyPipe::expectNoConsumption() + abortedConsumption = true; + + // in case somebody enabled auto-consumption before regular one aborted +- if (mustAutoConsume) +- startAutoConsumption(); ++ startAutoConsumptionIfNeeded(); + } + } + +@@ -313,23 +312,28 @@ BodyPipe::consume(size_t size) + postConsume(size); + } + +-// In the AutoConsumption mode the consumer has gone but the producer continues +-// producing data. We are using a BodySink BodyConsumer which just discards the produced data. + void + BodyPipe::enableAutoConsumption() + { + mustAutoConsume = true; + debugs(91,5, HERE << "enabled auto consumption" << status()); +- if (!theConsumer && theBuf.hasContent()) +- startAutoConsumption(); ++ startAutoConsumptionIfNeeded(); + } + +-// start auto consumption by creating body sink ++/// Check the current need and, if needed, start auto consumption. In auto ++/// consumption mode, the consumer is gone, but the producer continues to ++/// produce data. We use a BodySink BodyConsumer to discard that data. + void +-BodyPipe::startAutoConsumption() ++BodyPipe::startAutoConsumptionIfNeeded() + { +- Must(mustAutoConsume); +- Must(!theConsumer); ++ const auto startNow = ++ mustAutoConsume && // was enabled ++ !theConsumer && // has not started yet ++ theProducer.valid() && // still useful (and will eventually stop) ++ theBuf.hasContent(); // has something to consume right now ++ if (!startNow) ++ return; ++ + theConsumer = new BodySink(this); + debugs(91,7, HERE << "starting auto consumption" << status()); + scheduleBodyDataNotification(); +@@ -393,15 +397,14 @@ BodyPipe::postAppend(size_t size) + thePutSize += size; + debugs(91,7, HERE << "added " << size << " bytes" << status()); + +- if (mustAutoConsume && !theConsumer && size > 0) +- startAutoConsumption(); ++ if (!mayNeedMoreData()) ++ clearProducer(true); // reached end-of-body + + // We should not consume here even if mustAutoConsume because the + // caller may not be ready for the data to be consumed during this call. + scheduleBodyDataNotification(); + +- if (!mayNeedMoreData()) +- clearProducer(true); // reached end-of-body ++ startAutoConsumptionIfNeeded(); + } + + void +diff --git a/src/BodyPipe.h b/src/BodyPipe.h +index 2bf3220..7061c78 100644 +--- a/src/BodyPipe.h ++++ b/src/BodyPipe.h +@@ -131,7 +131,7 @@ public: + bool exhausted() const; // saw eof/abort and all data consumed + bool stillConsuming(const Consumer::Pointer &consumer) const { return theConsumer == consumer; } + +- // start or continue consuming when there is no consumer ++ /// start or continue consuming when producing without consumer + void enableAutoConsumption(); + + const MemBuf &buf() const { return theBuf; } +@@ -151,7 +151,7 @@ protected: + void postConsume(size_t size); + void postAppend(size_t size); + +- void startAutoConsumption(); // delayed start of enabled consumption ++ void startAutoConsumptionIfNeeded(); + + private: + int64_t theBodySize; // expected total content length, if known +@@ -163,7 +163,7 @@ private: + + MemBuf theBuf; // produced but not yet consumed content, if any + +- bool mustAutoConsume; // consume when there is no consumer ++ bool mustAutoConsume; ///< keep theBuf empty when producing without consumer + bool abortedConsumption; ///< called BodyProducer::noteBodyConsumerAborted + bool isCheckedOut; // to keep track of checkout violations + }; diff --git a/src/patches/squid/04_Fix_OpenSSL_builds_that_define_OPENSSL_NO_ENGINE_349.patch b/src/patches/squid/04_Fix_OpenSSL_builds_that_define_OPENSSL_NO_ENGINE_349.patch new file mode 100644 index 000000000..c868d585c --- /dev/null +++ b/src/patches/squid/04_Fix_OpenSSL_builds_that_define_OPENSSL_NO_ENGINE_349.patch @@ -0,0 +1,22 @@ +commit 186fc1d7c3bad845882d8e3497af9c3a42d75e8f +Author: Rosen Penev <rosenp(a)gmail.com> +Date: 2019-01-09 17:42:21 +0000 + + Fix OpenSSL builds that define OPENSSL_NO_ENGINE (#349) + + Even with ENGINE support disabled, OpenSSL provides the openssl/engine.h + header. We have to use OPENSSL_NO_ENGINE to detect ENGINE support. + +diff --git a/src/ssl/support.cc b/src/ssl/support.cc +index f6d4ce7..e350a73 100644 +--- a/src/ssl/support.cc ++++ b/src/ssl/support.cc +@@ -485,7 +485,7 @@ Ssl::Initialize(void) + + SQUID_OPENSSL_init_ssl(); + +-#if HAVE_OPENSSL_ENGINE_H ++#if !defined(OPENSSL_NO_ENGINE) + if (::Config.SSL.ssl_engine) { + ENGINE_load_builtin_engines(); + ENGINE *e; diff --git a/src/patches/squid/05_Fixed_disker-to-worker_queue_overflows_353.patch b/src/patches/squid/05_Fixed_disker-to-worker_queue_overflows_353.patch new file mode 100644 index 000000000..1b656549f --- /dev/null +++ b/src/patches/squid/05_Fixed_disker-to-worker_queue_overflows_353.patch @@ -0,0 +1,92 @@ +commit 4ce035593f5257ee2deb6ca85c7b9a0c4b850f70 +Author: Eduard Bagdasaryan <eduard.bagdasaryan(a)measurement-factory.com> +Date: 2019-01-12 10:05:30 +0000 + + Fixed disker-to-worker queue overflows (#353) + + Before this fix, Squid sometimes logged the following error: + + BUG: Worker I/O pop queue for ... overflow: ... + + The bug could result in truncated hit responses, reduced hit ratio, and, + combined with buggy lost I/O handling code (GitHub PR #352), even cache + corruption. + + The bug could be triggered by the following sequence of events: + + * Disker dequeues one I/O request from the worker push queue. + * Worker pushes more I/O requests to that disker, reaching 1024 requests + in its push queue (QueueCapacity or just "N" below). No overflow here! + * Worker process is suspended (or is just too busy to pop I/O results). + * Disker satisfies all 1+N requests, adding each to the worker pop queue + and overflows that queue when adding the last processed request. + + This fix limits worker push so that the sum of all pending requests + never exceeds (pop) queue capacity. This approach will continue to work + even if diskers are enhanced to dequeue multiple requests for seek + optimization and/or priority-based scheduling. + + Pop queue and push queue can still accommodate N requests each. The fix + appears to reduce supported disker "concurrency" levels from 2N down to + N pending I/O requests, reducing queue memory utilization. However, the + actual reduction is from N+1 to N: Since a worker pops all its satisfied + requests before queuing a new one, there could never be more than N+1 + pending requests (N in the push queue and 1 worked on by the disker). + + We left the BUG reporting and handling intact. There are no known bugs + in that code now. If the bug never surfaces again, it can be replaced + with code that translates low-level queue overflow exception into a + user-friendly TextException. + +diff --git a/src/DiskIO/IpcIo/IpcIoFile.cc b/src/DiskIO/IpcIo/IpcIoFile.cc +index 735cf0f..95d6d8b 100644 +--- a/src/DiskIO/IpcIo/IpcIoFile.cc ++++ b/src/DiskIO/IpcIo/IpcIoFile.cc +@@ -364,6 +364,10 @@ IpcIoFile::push(IpcIoPendingRequest *const pending) + + debugs(47, 7, HERE << "pushing " << SipcIo(KidIdentifier, ipcIo, diskId)); + ++ // protect DiskerHandleRequest() from pop queue overflow ++ if (pendingRequests() >= QueueCapacity) ++ throw Ipc::OneToOneUniQueue::Full(); ++ + if (queue->push(diskId, ipcIo)) + Notify(diskId); // must notify disker + trackPendingRequest(ipcIo.requestId, pending); +@@ -879,10 +883,8 @@ IpcIoFile::DiskerHandleRequest(const int workerId, IpcIoMsg &ipcIo) + if (queue->push(workerId, ipcIo)) + Notify(workerId); // must notify worker + } catch (const Queue::Full &) { +- // The worker queue should not overflow because the worker should pop() +- // before push()ing and because if disker pops N requests at a time, +- // we should make sure the worker pop() queue length is the worker +- // push queue length plus N+1. XXX: implement the N+1 difference. ++ // The worker pop queue should not overflow because the worker can ++ // push only if pendingRequests() is less than QueueCapacity. + debugs(47, DBG_IMPORTANT, "BUG: Worker I/O pop queue for " << + DbName << " overflow: " << + SipcIo(workerId, ipcIo, KidIdentifier)); // TODO: report queue len +diff --git a/src/DiskIO/IpcIo/IpcIoFile.h b/src/DiskIO/IpcIo/IpcIoFile.h +index f89197e..53a3749 100644 +--- a/src/DiskIO/IpcIo/IpcIoFile.h ++++ b/src/DiskIO/IpcIo/IpcIoFile.h +@@ -55,6 +55,8 @@ public: + + class IpcIoPendingRequest; + ++/// In a worker process, represents a single (remote) cache_dir disker file. ++/// In a disker process, used as a bunch of static methods handling that file. + class IpcIoFile: public DiskFile + { + CBDATA_CLASS(IpcIoFile); +@@ -98,6 +100,10 @@ private: + void push(IpcIoPendingRequest *const pending); + IpcIoPendingRequest *dequeueRequest(const unsigned int requestId); + ++ /// the total number of I/O requests in push queue and pop queue ++ /// (but no, the implementation does not add push and pop queue sizes) ++ size_t pendingRequests() const { return olderRequests->size() + newerRequests->size(); } ++ + static void Notify(const int peerId); + + static void OpenTimeout(void *const param); diff --git a/src/patches/squid/06_Initialize_StoreMapSlice_when_reserving_a_new_cache_slot_350.patch b/src/patches/squid/06_Initialize_StoreMapSlice_when_reserving_a_new_cache_slot_350.patch new file mode 100644 index 000000000..5f04bffce --- /dev/null +++ b/src/patches/squid/06_Initialize_StoreMapSlice_when_reserving_a_new_cache_slot_350.patch @@ -0,0 +1,246 @@ +commit 8e022a61922053764fd1627ba4b65a1617a1bd8a +Author: Eduard Bagdasaryan <eduard.bagdasaryan(a)measurement-factory.com> +Date: 2019-01-23 03:51:12 +0000 + + Initialize StoreMapSlice when reserving a new cache slot (#350) + + Rock sets the StoreMapSlice::next field when sending a slice to disk. To + avoid writing slice A twice, Rock allocates a new slice B to prime + A.next right before writing A. Scheduling A's writing and, sometimes, + lack of data to fill B create a gap between B's allocation and B's + writing (which sets B.next). During that time, A.next points to B, but + B.next is untouched. + + If writing slice A or swapout in general fails, the chain of failed + entry slices (now containing both A and B) is freed. If untouched B.next + contains garbage, then freeChainAt() adds "random" slices after B to the + free slice pool. Subsequent swapouts use those incorrectly freed slices, + effectively overwriting portions of random cache entries, corrupting the + cache. + + How did B.next get dirty in the first place? freeChainAt() cleans the + slices it frees, but Rock also makes direct noteFreeMapSlice() calls. + Shared memory cache may have avoided this corruption because it makes no + such calls. + + Ipc::StoreMap::prepFreeSlice() now clears allocated slices. Long-term, + we may be able to move free slice management into StoreMap to automate + this cleanup. + + Also simplified and polished slot allocation code a little, removing the + Rock::IoState::reserveSlotForWriting() middleman. This change also + improves the symmetry between Rock and shared memory cache code. + +diff --git a/src/MemStore.cc b/src/MemStore.cc +index 1b24469..1ff48a0 100644 +--- a/src/MemStore.cc ++++ b/src/MemStore.cc +@@ -772,13 +772,15 @@ MemStore::reserveSapForWriting(Ipc::Mem::PageId &page) + { + Ipc::Mem::PageId slot; + if (freeSlots->pop(slot)) { +- debugs(20, 5, "got a previously free slot: " << slot); ++ const auto slotId = slot.number - 1; ++ debugs(20, 5, "got a previously free slot: " << slotId); + + if (Ipc::Mem::GetPage(Ipc::Mem::PageId::cachePage, page)) { + debugs(20, 5, "and got a previously free page: " << page); +- return slot.number - 1; ++ map->prepFreeSlice(slotId); ++ return slotId; + } else { +- debugs(20, 3, "but there is no free page, returning " << slot); ++ debugs(20, 3, "but there is no free page, returning " << slotId); + freeSlots->push(slot); + } + } +@@ -791,8 +793,10 @@ MemStore::reserveSapForWriting(Ipc::Mem::PageId &page) + assert(!waitingFor); // noteFreeMapSlice() should have cleared it + assert(slot.set()); + assert(page.set()); +- debugs(20, 5, "got previously busy " << slot << " and " << page); +- return slot.number - 1; ++ const auto slotId = slot.number - 1; ++ map->prepFreeSlice(slotId); ++ debugs(20, 5, "got previously busy " << slotId << " and " << page); ++ return slotId; + } + assert(waitingFor.slot == &slot && waitingFor.page == &page); + waitingFor.slot = NULL; +diff --git a/src/fs/rock/RockIoState.cc b/src/fs/rock/RockIoState.cc +index bba24ae..50b6e6d 100644 +--- a/src/fs/rock/RockIoState.cc ++++ b/src/fs/rock/RockIoState.cc +@@ -190,7 +190,7 @@ Rock::IoState::tryWrite(char const *buf, size_t size, off_t coreOff) + // allocate the first slice during the first write + if (!coreOff) { + assert(sidCurrent < 0); +- sidCurrent = reserveSlotForWriting(); // throws on failures ++ sidCurrent = dir->reserveSlotForWriting(); // throws on failures + assert(sidCurrent >= 0); + writeAnchor().start = sidCurrent; + } +@@ -207,7 +207,7 @@ Rock::IoState::tryWrite(char const *buf, size_t size, off_t coreOff) + // We do not write a full buffer without overflow because + // we would not yet know what to set the nextSlot to. + if (overflow) { +- const SlotId sidNext = reserveSlotForWriting(); // throws ++ const auto sidNext = dir->reserveSlotForWriting(); // throws + assert(sidNext >= 0); + writeToDisk(sidNext); + } else if (Store::Root().transientReaders(*e)) { +@@ -306,21 +306,6 @@ Rock::IoState::writeBufToDisk(const SlotId sidNext, const bool eof, const bool l + theFile->write(r); + } + +-/// finds and returns a free db slot to fill or throws +-Rock::SlotId +-Rock::IoState::reserveSlotForWriting() +-{ +- Ipc::Mem::PageId pageId; +- if (dir->useFreeSlot(pageId)) +- return pageId.number-1; +- +- // This may happen when the number of available db slots is close to the +- // number of concurrent requests reading or writing those slots, which may +- // happen when the db is "small" compared to the request traffic OR when we +- // are rebuilding and have not loaded "many" entries or empty slots yet. +- throw TexcHere("ran out of free db slots"); +-} +- + void + Rock::IoState::finishedWriting(const int errFlag) + { +diff --git a/src/fs/rock/RockIoState.h b/src/fs/rock/RockIoState.h +index 3c21355..b8f2061 100644 +--- a/src/fs/rock/RockIoState.h ++++ b/src/fs/rock/RockIoState.h +@@ -66,7 +66,6 @@ private: + size_t writeToBuffer(char const *buf, size_t size); + void writeToDisk(const SlotId nextSlot); + void writeBufToDisk(const SlotId nextSlot, const bool eof, const bool lastWrite); +- SlotId reserveSlotForWriting(); + + void callBack(int errflag); + +diff --git a/src/fs/rock/RockSwapDir.cc b/src/fs/rock/RockSwapDir.cc +index 06fb763..f170bb6 100644 +--- a/src/fs/rock/RockSwapDir.cc ++++ b/src/fs/rock/RockSwapDir.cc +@@ -699,12 +699,16 @@ Rock::SwapDir::diskOffsetLimit() const + return diskOffset(map->sliceLimit()); + } + +-bool +-Rock::SwapDir::useFreeSlot(Ipc::Mem::PageId &pageId) ++Rock::SlotId ++Rock::SwapDir::reserveSlotForWriting() + { ++ Ipc::Mem::PageId pageId; ++ + if (freeSlots->pop(pageId)) { +- debugs(47, 5, "got a previously free slot: " << pageId); +- return true; ++ const auto slotId = pageId.number - 1; ++ debugs(47, 5, "got a previously free slot: " << slotId); ++ map->prepFreeSlice(slotId); ++ return slotId; + } + + // catch free slots delivered to noteFreeMapSlice() +@@ -713,14 +717,20 @@ Rock::SwapDir::useFreeSlot(Ipc::Mem::PageId &pageId) + if (map->purgeOne()) { + assert(!waitingForPage); // noteFreeMapSlice() should have cleared it + assert(pageId.set()); +- debugs(47, 5, "got a previously busy slot: " << pageId); +- return true; ++ const auto slotId = pageId.number - 1; ++ debugs(47, 5, "got a previously busy slot: " << slotId); ++ map->prepFreeSlice(slotId); ++ return slotId; + } + assert(waitingForPage == &pageId); + waitingForPage = NULL; + ++ // This may happen when the number of available db slots is close to the ++ // number of concurrent requests reading or writing those slots, which may ++ // happen when the db is "small" compared to the request traffic OR when we ++ // are rebuilding and have not loaded "many" entries or empty slots yet. + debugs(47, 3, "cannot get a slot; entries: " << map->entryCount()); +- return false; ++ throw TexcHere("ran out of free db slots"); + } + + bool +diff --git a/src/fs/rock/RockSwapDir.h b/src/fs/rock/RockSwapDir.h +index 879bd45..aa88283 100644 +--- a/src/fs/rock/RockSwapDir.h ++++ b/src/fs/rock/RockSwapDir.h +@@ -62,10 +62,12 @@ public: + int64_t slotLimitAbsolute() const; ///< Rock store implementation limit + int64_t slotLimitActual() const; ///< total number of slots in this db + +- /// removes a slot from a list of free slots or returns false +- bool useFreeSlot(Ipc::Mem::PageId &pageId); + /// whether the given slot ID may point to a slot in this db + bool validSlotId(const SlotId slotId) const; ++ ++ /// finds and returns a free db slot to fill or throws ++ SlotId reserveSlotForWriting(); ++ + /// purges one or more entries to make full() false and free some slots + void purgeSome(); + +diff --git a/src/ipc/StoreMap.cc b/src/ipc/StoreMap.cc +index ba1d4df..e778bba 100644 +--- a/src/ipc/StoreMap.cc ++++ b/src/ipc/StoreMap.cc +@@ -346,8 +346,7 @@ Ipc::StoreMap::freeChainAt(SliceId sliceId, const SliceId splicingPoint) + while (sliceId >= 0) { + Slice &slice = sliceAt(sliceId); + const SliceId nextId = slice.next; +- slice.size = 0; +- slice.next = -1; ++ slice.clear(); + if (cleaner) + cleaner->noteFreeMapSlice(sliceId); // might change slice state + if (sliceId == splicingPoint) { +@@ -360,6 +359,14 @@ Ipc::StoreMap::freeChainAt(SliceId sliceId, const SliceId splicingPoint) + debugs(54, 7, "freed chain #" << chainId << " in " << path); + } + ++void ++Ipc::StoreMap::prepFreeSlice(const SliceId sliceId) ++{ ++ // TODO: Move freeSlots here, along with reserveSlotForWriting() logic. ++ assert(validSlice(sliceId)); ++ sliceAt(sliceId).clear(); ++} ++ + Ipc::StoreMap::SliceId + Ipc::StoreMap::sliceContaining(const sfileno fileno, const uint64_t bytesNeeded) const + { +diff --git a/src/ipc/StoreMap.h b/src/ipc/StoreMap.h +index 5d44379..8a4909e 100644 +--- a/src/ipc/StoreMap.h ++++ b/src/ipc/StoreMap.h +@@ -42,6 +42,9 @@ public: + return *this; + } + ++ /// restore default-constructed state ++ void clear() { size = 0; next = -1; } ++ + std::atomic<Size> size; ///< slice contents size + std::atomic<StoreMapSliceId> next; ///< ID of the next entry slice + }; +@@ -292,6 +295,9 @@ public: + /// readable anchor for the entry created by openForReading() + const Anchor &readableEntry(const AnchorId anchorId) const; + ++ /// prepare a chain-unaffiliated slice for being added to an entry chain ++ void prepFreeSlice(const SliceId sliceId); ++ + /// Returns the ID of the entry slice containing n-th byte or + /// a negative ID if the entry does not store that many bytes (yet). + /// Requires a read lock. diff --git a/src/patches/squid/07_Fail_Rock_swapout_if_the_disk_dropped_some_of_the_write_requests_352.patch b/src/patches/squid/07_Fail_Rock_swapout_if_the_disk_dropped_some_of_the_write_requests_352.patch new file mode 100644 index 000000000..14ce96a33 --- /dev/null +++ b/src/patches/squid/07_Fail_Rock_swapout_if_the_disk_dropped_some_of_the_write_requests_352.patch @@ -0,0 +1,164 @@ +commit a374216b3262b6b40dc19b8ea5423cd2f27f94a3 +Author: Eduard Bagdasaryan <eduard.bagdasaryan(a)measurement-factory.com> +Date: 2019-01-23 04:30:40 +0000 + + Fail Rock swapout if the disk dropped some of the write requests (#352) + + Detecting dropped writes earlier is more than a TODO: If the last entry + write was successful, the whole entry becomes available for hits + immediately. IpcIoFile::checkTimeouts() that runs every 7 seconds + (IpcIoFile::Timeout) would eventually notify Rock about the timeout, + allowing Rock to release the failed entry, but that notification may + be too late. + + The precise outcome of hitting an entry with a missing on-disk slice is + unknown (because the bug was detected by temporary hit validation code + that turned such hits into misses), but SWAPFAIL is the best we could + hope for. + +diff --git a/src/fs/rock/RockSwapDir.cc b/src/fs/rock/RockSwapDir.cc +index f170bb6..b70dc8b 100644 +--- a/src/fs/rock/RockSwapDir.cc ++++ b/src/fs/rock/RockSwapDir.cc +@@ -858,42 +858,59 @@ Rock::SwapDir::writeCompleted(int errflag, size_t, RefCount< ::WriteRequest> r) + + debugs(79, 7, "errflag=" << errflag << " rlen=" << request->len << " eof=" << request->eof); + +- // TODO: Fail if disk dropped one of the previous write requests. +- +- if (errflag == DISK_OK) { +- // do not increment sio.offset_ because we do it in sio->write() +- +- // finalize the shared slice info after writing slice contents to disk +- Ipc::StoreMap::Slice &slice = +- map->writeableSlice(sio.swap_filen, request->sidCurrent); +- slice.size = request->len - sizeof(DbCellHeader); +- slice.next = request->sidNext; +- +- if (request->eof) { +- assert(sio.e); +- assert(sio.writeableAnchor_); +- if (sio.touchingStoreEntry()) { +- sio.e->swap_file_sz = sio.writeableAnchor_->basics.swap_file_sz = +- sio.offset_; +- map->switchWritingToReading(sio.swap_filen); +- // sio.e keeps the (now read) lock on the anchor +- } +- sio.writeableAnchor_ = NULL; +- sio.splicingPoint = request->sidCurrent; +- sio.finishedWriting(errflag); +- } +- } else { +- noteFreeMapSlice(request->sidNext); +- +- writeError(sio); +- sio.finishedWriting(errflag); +- // and wait for the finalizeSwapoutFailure() call to close the map entry +- } ++ if (errflag != DISK_OK) ++ handleWriteCompletionProblem(errflag, *request); ++ else if (droppedEarlierRequest(*request)) ++ handleWriteCompletionProblem(DISK_ERROR, *request); ++ else ++ handleWriteCompletionSuccess(*request); + + if (sio.touchingStoreEntry()) + CollapsedForwarding::Broadcast(*sio.e); + } + ++/// code shared by writeCompleted() success handling cases ++void ++Rock::SwapDir::handleWriteCompletionSuccess(const WriteRequest &request) ++{ ++ auto &sio = *(request.sio); ++ sio.splicingPoint = request.sidCurrent; ++ // do not increment sio.offset_ because we do it in sio->write() ++ ++ // finalize the shared slice info after writing slice contents to disk ++ Ipc::StoreMap::Slice &slice = ++ map->writeableSlice(sio.swap_filen, request.sidCurrent); ++ slice.size = request.len - sizeof(DbCellHeader); ++ slice.next = request.sidNext; ++ ++ if (request.eof) { ++ assert(sio.e); ++ assert(sio.writeableAnchor_); ++ if (sio.touchingStoreEntry()) { ++ sio.e->swap_file_sz = sio.writeableAnchor_->basics.swap_file_sz = ++ sio.offset_; ++ ++ map->switchWritingToReading(sio.swap_filen); ++ // sio.e keeps the (now read) lock on the anchor ++ } ++ sio.writeableAnchor_ = NULL; ++ sio.finishedWriting(DISK_OK); ++ } ++} ++ ++/// code shared by writeCompleted() error handling cases ++void ++Rock::SwapDir::handleWriteCompletionProblem(const int errflag, const WriteRequest &request) ++{ ++ auto &sio = *request.sio; ++ ++ noteFreeMapSlice(request.sidNext); ++ ++ writeError(sio); ++ sio.finishedWriting(errflag); ++ // and hope that Core will call disconnect() to close the map entry ++} ++ + void + Rock::SwapDir::writeError(StoreIOState &sio) + { +@@ -908,6 +925,23 @@ Rock::SwapDir::writeError(StoreIOState &sio) + // All callers must also call IoState callback, to propagate the error. + } + ++/// whether the disk has dropped at least one of the previous write requests ++bool ++Rock::SwapDir::droppedEarlierRequest(const WriteRequest &request) const ++{ ++ const auto &sio = *request.sio; ++ assert(sio.writeableAnchor_); ++ const Ipc::StoreMapSliceId expectedSliceId = sio.splicingPoint < 0 ? ++ sio.writeableAnchor_->start : ++ map->writeableSlice(sio.swap_filen, sio.splicingPoint).next; ++ if (expectedSliceId != request.sidCurrent) { ++ debugs(79, 3, "yes; expected " << expectedSliceId << ", but got " << request.sidCurrent); ++ return true; ++ } ++ ++ return false; ++} ++ + void + Rock::SwapDir::updateHeaders(StoreEntry *updatedE) + { +diff --git a/src/fs/rock/RockSwapDir.h b/src/fs/rock/RockSwapDir.h +index aa88283..0748337 100644 +--- a/src/fs/rock/RockSwapDir.h ++++ b/src/fs/rock/RockSwapDir.h +@@ -138,6 +138,9 @@ protected: + + private: + void createError(const char *const msg); ++ void handleWriteCompletionSuccess(const WriteRequest &request); ++ void handleWriteCompletionProblem(const int errflag, const WriteRequest &request); ++ bool droppedEarlierRequest(const WriteRequest &request) const; + + DiskIOStrategy *io; + RefCount<DiskFile> theFile; ///< cache storage for this cache_dir +diff --git a/src/fs/rock/forward.h b/src/fs/rock/forward.h +index ba64d03..4e68a27 100644 +--- a/src/fs/rock/forward.h ++++ b/src/fs/rock/forward.h +@@ -40,6 +40,8 @@ class HeaderUpdater; + + class DbCellHeader; + ++class WriteRequest; ++ + } + + #endif /* SQUID_FS_ROCK_FORWARD_H */ diff --git a/src/patches/squid/08_Bug_4914_Do_not_call_setsid_in_--foreground_mode_354.patch b/src/patches/squid/08_Bug_4914_Do_not_call_setsid_in_--foreground_mode_354.patch new file mode 100644 index 000000000..65bf2e648 --- /dev/null +++ b/src/patches/squid/08_Bug_4914_Do_not_call_setsid_in_--foreground_mode_354.patch @@ -0,0 +1,36 @@ +commit 6076ca90a8101888483b769edd2f9aeef0da8638 +Author: Marcos Mello <marcosfrm(a)users.noreply.github.com> +Date: 2019-01-23 15:18:43 +0000 + + Bug 4914: Do not call setsid() in --foreground mode (#354) + + Squid executed in --foreground is always a process group leader. Thus, + setsid(2) is unnecessary and always fails (with EPERM) for such Squids. + +diff --git a/src/main.cc b/src/main.cc +index 7f5ec80..1e8e035 100644 +--- a/src/main.cc ++++ b/src/main.cc +@@ -1871,6 +1871,7 @@ GoIntoBackground() + exit(EXIT_SUCCESS); + } + // child, running as a background daemon ++ Must(setsid() > 0); // ought to succeed after fork() + } + + static void +@@ -1916,14 +1917,6 @@ watch_child(const CommandLine &masterCommand) + if (!opt_foreground) + GoIntoBackground(); + +- // TODO: Fails with --foreground if the calling process is process group +- // leader, which is always (?) the case. Should probably moved to +- // GoIntoBackground and executed only after successfully forking +- if (setsid() < 0) { +- int xerrno = errno; +- syslog(LOG_ALERT, "setsid failed: %s", xstrerr(xerrno)); +- } +- + closelog(); + + #ifdef TIOCNOTTY diff --git a/src/patches/squid/09_Bug_4915_Detect_IPv6_loopback_binding_errors_355.patch b/src/patches/squid/09_Bug_4915_Detect_IPv6_loopback_binding_errors_355.patch new file mode 100644 index 000000000..b6fed3336 --- /dev/null +++ b/src/patches/squid/09_Bug_4915_Detect_IPv6_loopback_binding_errors_355.patch @@ -0,0 +1,54 @@ +commit 568e66b7c64e0d7873b9575b0dc60886805a2fc6 (refs/remotes/origin/v4) +Author: Amos Jeffries <yadij(a)users.noreply.github.com> +Date: 2019-01-24 16:43:47 +0000 + + Bug 4915: Detect IPv6 loopback binding errors (#355) + + Systems which have been partially 'IPv6 disabled' may allow + sockets to be opened and used but missing the IPv6 loopback + address. + + Implement the outstanding TODO to detect such failures and + disable IPv6 support properly within Squid when they are found. + + This should fix bug 4915 auth_param helper startup and similar + external_acl_type helper issues. For security such helpers are + not permitted to use the machine default IP address which is + globally accessible. + +diff --git a/src/ip/tools.cc b/src/ip/tools.cc +index c7a8ac8..1ccf64f 100644 +--- a/src/ip/tools.cc ++++ b/src/ip/tools.cc +@@ -10,6 +10,7 @@ + + #include "squid.h" + #include "Debug.h" ++#include "ip/Address.h" + #include "ip/tools.h" + + #if HAVE_UNISTD_H +@@ -55,8 +56,21 @@ Ip::ProbeTransport() + debugs(3, 2, "Missing RFC 3493 compliance - attempting split IPv4 and IPv6 stacks ..."); + EnableIpv6 |= IPV6_SPECIAL_SPLITSTACK; + #endif +- // TODO: attempt to use the socket to connect somewhere ? +- // needs to be safe to contact and with guaranteed working IPv6 at the other end. ++ ++ // Test for IPv6 loopback/localhost address binding ++ Ip::Address ip; ++ ip.setLocalhost(); ++ if (ip.isIPv6()) { // paranoid; always succeeds if we got this far ++ struct sockaddr_in6 sin; ++ ip.getSockAddr(sin); ++ if (bind(s, reinterpret_cast<struct sockaddr *>(&sin), sizeof(sin)) != 0) { ++ debugs(3, DBG_CRITICAL, "WARNING: BCP 177 violation. Detected non-functional IPv6 loopback."); ++ EnableIpv6 = IPV6_OFF; ++ } else { ++ debugs(3, 2, "Detected functional IPv6 loopback ..."); ++ } ++ } ++ + close(s); + + #if USE_IPV6 -- 2.18.0

2 7

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Development February 2019