Update of Suricata + HPFeeds support#1
Open
Philelis wants to merge 2554 commits into
Open
Conversation
If the app-layer-parsing has a very long content it exceeds the maximum defined in "alproto_name". This adds a check for the too long content before it will be passed to "strlcpy" and logs an error.
The JSON logger had already been updated to handle transactions without a response. Apply the same logic to the older dns-log where a logger is registered for each direction. Fixes issue 2012.
When packet is coming from a real ethernet card, the kernel is stripping the vlan header and delivering a modified packet so we need to insert the VLAN header back before sending the packet on the wire. To do so, we pass an option to the raw socket to add a reserve before the packet data. It will get Suricata some head room to to move the ethernet addresses before there actual place and and insert the VLAN header in the correct place. We get VLAN info from the ring buffer as the call of AFPWrite is always done in the release function so we still have access to the memory.
The code to get the rule group (sgh) would return the group for IP proto 0 instead of nothing. This lead to certain types of rules unintentionally matching (False Positive). Since the packets weren't actually IP, the logged alert records were missing the IP header. Bug OISF#2017.
The `ts_ecr' and `ts_val' struct fields are integer types, not pointers. This leads GCC 6.3.0 to complain about comparisons to NULL. Signed-off-by: Sascha Steinbiss <sascha@steinbiss.name>
The size of a memory buffer to be allocated was kept in a signed int instead of a size_t, leading to an overflow when large lists of long and diverse patterns cause the amount of AC states to blow up (>2GB). Fixes Redmine issues OISF#1827 and OISF#1843. Signed-off-by: Sascha Steinbiss <sascha@steinbiss.name>
Due to the use of AFL_LOOP and initialization/deinit outside of it, part of the fuzzing relied on the global 'state' in flow and defrag. Because of this crashes that were found could not be reproduced. The saved crash input was only the last in the series. This patch addresses that. It requires a new output directory 'dump' where the packet fuzzers will store all their input. If the AFL_LOOP fails the files will not be removed and this 'serie' can be read again for reproducing the issue. e.g.: AFL would work with: --afl-decoder-ppp=@@ and after a crash is found the produced serie can be read with: --afl-decoder-ppp-serie=1486656919-514163 The series have a timestamp as name and a suffix that controls the order in which the files will be 'replayed' in Suricata.
in suricata source dir: mkdir afl cd afl bash ../scripts/afl/runafl.sh decoder-ipv4
The IP protocol was not being used to match fragments with their packets allowing a carefully constructed packet with a different protocol to be matched, allowing re-assembly to complete, creating a packet that would not be re-assembled by the destination host.
On a zero size A or AAAA record, 4 or 16 bytes would still be read. Found with AFL+ASAN.
It was already marked as depricated and no longer in use anywhere.
clang-4.0 reported:
util-pool.c:242:13: warning: logical not is only applied to the left hand side of this bitwise operator [-Wlogical-not-parentheses]
if (! pb->flags & POOL_BUCKET_PREALLOCATED) {
^ ~
util-pool.c:242:13: note: add parentheses after the '!' to evaluate the bitwise operator first
if (! pb->flags & POOL_BUCKET_PREALLOCATED) {
^
( )
util-pool.c:242:13: note: add parentheses around left hand side expression to silence this warning
if (! pb->flags & POOL_BUCKET_PREALLOCATED) {
^
( )
util-pool.c:261:13: warning: logical not is only applied to the left hand side of this bitwise operator [-Wlogical-not-parentheses]
if (! pb->flags & POOL_BUCKET_PREALLOCATED) {
^ ~
util-pool.c:261:13: note: add parentheses after the '!' to evaluate the bitwise operator first
if (! pb->flags & POOL_BUCKET_PREALLOCATED) {
^
( )
util-pool.c:261:13: note: add parentheses around left hand side expression to silence this warning
if (! pb->flags & POOL_BUCKET_PREALLOCATED) {
^
( )
2 warnings generated.
app-layer-modbus.c:1226:39: warning: taking address of packed member 'transactionId' of class or structure 'ModbusHeader_' may result in an unaligned pointer value [-Waddress-of-packed-member]
if (ModbusExtractUint16(modbus, &(header->transactionId), input, input_len, &offset) ||
^~~~~~~~~~~~~~~~~~~~~
app-layer-modbus.c:1228:39: warning: taking address of packed member 'protocolId' of class or structure 'ModbusHeader_' may result in an unaligned pointer value [-Waddress-of-packed-member]
ModbusExtractUint16(modbus, &(header->protocolId), input, input_len, &offset) ||
^~~~~~~~~~~~~~~~~~
app-layer-modbus.c:1230:39: warning: taking address of packed member 'length' of class or structure 'ModbusHeader_' may result in an unaligned pointer value [-Waddress-of-packed-member]
ModbusExtractUint16(modbus, &(header->length), input, input_len, &offset) ||
^~~~~~~~~~~~~~
3 warnings generated.
Bug OISF#2088
detect-geoip.c:78:40: error: incompatible pointer types assigning to 'int (*)(ThreadVars *, DetectEngineThreadCtx *, Packet *, const Signature *, const SigMatchCtx *)' from 'int (ThreadVars *, DetectEngineThreadCtx *, Packet *, Signature *, const SigMatchCtx *)' [-Werror,-Wincompatible-pointer-types]
sigmatch_table[DETECT_GEOIP].Match = DetectGeoipMatch;
^ ~~~~~~~~~~~~~~~~
1 error generated.
Bug OISF#2045
UnixManagerThreadInit needs to return a failure code if the socket fails to initialize to avoid entering the UnixManager loop which will continuously fail on the call to bind, as no listening socket was setup. This can occur when the socket fails to initialize due to a permissions error and fatal init errors is not on.
Migrate to the new Travis container build system. This build system does not allow use of sudo, so required packages must be done declaratively which required reworking how we install packages that are conditional based on the build being done. Mac builds are still done with sudo=true.
Also reformat unit test functions to Suricata style.
If a subsequent fragment has a lower offset than a previous one and overlaps, trim off the beginning of the previous fragment. Based on an issue reported privately.
It appears that both using gcc and clang something gets misoptimised around pcre's jit. So disable jit for now.
Allow log filenames to contain date modifiers, e.g.:
- eve-log:
filename: eve-%Y-%m-%d-%H:%M:%S.json
Rotate log file based on time. Support both rotating based on a timer (XXs, XXm, XXd, XXw) and rotating based on a absolute time, like each minute, hour or day.
Recursively create new log directories when needed. This makes it possible to use date modifiers in the file path to create directories based on date, e.g.: /var/log/suricata/2017/02/14/
Writing to a unix socket can cause Suricata to block in the packet path. This could happen if the read-endpoint of the unix socket stays connected, but stops reading, or simply can't read fast enough as part of its event processing. To choose packets over events, do non-blocking socket writes and drop the event if the write would block and update a dropped counter.
If running against a pcap there is no reason to drop events, a blocking socket is fine here. So only do non-blocking writes when running off a live device.
Disable jit only for libpcre 8.39 and 8.40 as those were the buggy versions. Thanks to Zoltán Herczeg.
Check for ed was failing, as it was actually looking for edx.
Fix errors and simplify filters.
Use more explicit types instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I forked the last version of suricata and converted your hpfeeds solution.
Tested on CentOS 7