Sguil FAQ Table of Contents 1.0 Administrivia 1.1 Who is responsible for this FAQ? 1.2 I have a suggestion... 2.0 Sguil Basics 2.1 What is Sguil? 2.2 Can you describe the sguil architecture? 2.3 How is sguil different from Snort + ACID or Snort + BASE? 2.4 Who wrote sguil? 2.5 Where can I find more information/documentation about sguil? 2.6 Are there any sguil mailing lists? 2.7 What about IRC channels? 2.8 How can I try out sguil without installing it all? 2.9 I heard sguild is about to be rewritten in perl/C/Java/ADA... 2.10 Seriously. Why Tcl? 2.11 What is Network Security Monitoring (NSM)? 2.12 Why is it called "sguil"? 3.0 Installation and Configuration 3.1 Where do I get sguil? 3.2 Should I use the stable version or the CVS development release? 3.3 What other software does sguil depend on? 3.4 This is really complicated. Is there an installation and configuration guide? 3.5 How can I tell if it's all working right? 3.6 How does sguil capture network session information? 3.7 Should I use snort or sancp to capture session information? 4.0 Using Sguil 4.1 How do I set up sguil to automatically categorize incoming alerts? 4.2 Can sguil page me when it sees a particular alert? 5.0 Troubleshooting 5.1 Something's not working. Where do I start? 5.2 Putting sguild into debug mode 5.3 Putting sensor_agent into debug mode 5.4 I'm seeing IPC problems with sguild, and data isn't being loaded into the database 6.0 Miscellaneous 1.0 Administrivia 1.1 Who is responsible for this FAQ? In the immediate sense, that would be me, David Bianco, the FAQ editor. I use sguil at work, and thought that maintaining this FAQ would be a good way to give something back to the sguil community. In the larger sense, though, we are all responsible for the FAQ. It won't be very good unless people submit their own questions, and perhaps answers. If you'd like to contribute, see this question. 1.2 I have a suggestion... If you have a question or answer to contribute, or would just like to drop me a line to let me know what you think about the FAQ, please email me at "david at biancorama d0t com". No attachements, please. Plain text email is very much preferred. If I use your comments in the FAQ, I'll credit you, so be sure to let me know how you'd like to be listed. 2.0 Sguil Basics 2.1 What is Sguil? Sguil (pronounced sgweel) is probably best described as an aggregation system for network security monitoring tools. It ties your IDS alerts into a database of TCP/IP sessions, full content packet logs and other information. When you've identified an alert that needs more investigation, the sguil client provides you with seamless access to the data you need to decide how to handle the situation. In other words, sguil simply ties together the outputs of various security monitoring tools into a single interface, providing you with the most information in the shortest amount of time. sguil uses a database backend for most of its data, which allows you to perform SQL queries against several different types of security events. 2.2 Can you describe the sguil architecture? The following is from the sguil website: Sguil consist of three main components, a plugin to barnyard (op_sguil), a GUI server (sguild), and a GUI client (sguil.tk). A diagram defining how the different components interact with each other, is available here (vi-Sio is recommended for viewing). Once installed, these components allow the analyst to view snort events in near real time. Events can be validated by placing them into one of seven incident categories or marking the event as having no further action required (NA). These actions remove the events from the RealTime tab of all the connected clients but are not deleted from the database. Archived events can easily be retrieved from the database through preformatted queries, or the analyst can create a custom query using SQL. Also included in the sguil package, is a patch for the portscan preprocessor (logs to a pipe deliminated file), a patch for the stream4 preprocessor (keepstats type 'db' for logging pipe deliminated stats to a file), and a tcl script (sensor_agent.tcl) for loading the modified outputs into the database. These components give the analyst immediate access to portscan and session data. The final components are for analyzing the raw data associated with a given session. Xscriptd is a daemon that listens for request from sguil.tk and once queried, it parses raw tcpdump files for packets matching the requested session and either feeds the stream through tcpflow creating a transcript or sends the binary data back to the client to be loaded into ethereal. A sguil system is composed of a single sguil server and an arbitrary number of sguil network sensors. The sensors perform all the security monitoring tasks and feed information back to the server on a regular basis. The server coordinates this information, stores it in a database and communicates with sguil clients running on administrators' desktop machines. In ASCII terms, this looks like: |---------> Sensor #1 Client ==> sguil Server =========> |---------> Sensor #2 |---------> Sensor #3 Each sensor monitors a single network link (although you can have multiple sensors on one physical machine). They collect several different types of information: Snort monitors the link for security events, and logs them to a file on the local disk. Security events include not only IDS alerts but information about TCP/IP sessions and port scans. Barnyard takes events from the snort log file and inserts them into database running on the sguil server in near real-time A separate instance of snort logs the full content of all network packets to the local disk (this typically requires a large separate data partition) SANCP records TCP/IP sessions and fowards them to the database on the sguil server A special sguil agent listens for commands from the sguil server. These commands are typically requests for packet data previously logged by Snort. 2.3 How is sguil different from Snort + ACID or Snort + BASE? ACID & BASE are both web-based IDS alert management systems. They let you browse and search alerts, but don't attempt to offer very much in the way of data-mining that would allow you to answer questions like, "Was this an attack attempt or a false positive?", "Was the attempt successful?" or "What other machines did the attacker try to crack once he got into this one?". They pretty much rely on you to do the research necessary to determine the severity of the situation. Sguil's design centers on providing convenient, quick access to a host of supporting information, which both saves you time and helps you make better decisions. Incidentally, because sguil uses a dedicated client instead of running through a web browser, you get a richer, more responsive user interface as well. For a more complete answer to this question, I recommend Richard Bejtlich's post to the snort-users email list. 2.4 Who wrote sguil? Real working security analysts. We know your pain. Sguil's primary author is Bamm Visscher (AKA "qru" on IRC), but the entire sguil community contributes in one fashion or another. 2.5 Where can I find more information/documentation about sguil? Start with the sguil web page, http://www.sguil.org/. There you'll find screenshots, a Flash demo, installation documentation and more. Once you've digested that, I highly recommend Richard Bejtlich's The Tao of Network Security Monitoring. This book is the best general reference for the Network Security Monitoring (NSM) methodology, and the sguil chapter is available for free from Richard's website. 2.6 Are there any sguil mailing lists? Yes, there are three. The sguil-users (mirror) list is for general sguil questions, problems or comments. There's also a sguil-dev (mirror) list for developer discussions, and a sguil-cvs (mirror) list that tracks changes that have been checked into the source tree. The easiest way to subscribe to any of these lists is to use the subscription page. 2.7 What about IRC channels? The sguil community uses IRC as a primary day-to-day communication method, and anyone is welcome to join in the discussions. If you have an IRC client, connect to irc.freenode.net, then join #snort-gui. 2.8 How can I try out sguil without installing it all? You have two options. First, try the Flash demo of the sguil client in action. This is incomplete right now, but is certainly the easiest way to get a feel for how this all works. If you want a more interactive approach, you can download and install just the sguil client from this page. Once you've got it running, point the client at the demonstration server at demo.sguil.net on port 7734. You'll need to enter a username and password, but you can just make up anything. The client will ask what sensor you want to monitor (use the one called "reset"), and then bring up the main console window for you to play around with. Once you're connected, check out the "User Messages" pane on the main window. On the demonstration server, this gateways to the IRC channel so you can ask questions, get help or have a developer-led tour of the system. 2.9 I heard sguil is about to be rewritten in perl/C/Java/ADA... Yup, any day now. Honestly. 2.10 Seriously. Why Tcl? qru says: Um, cause I have a tcl book. :) 2.11 What is Network Security Monitoring (NSM)? NSM is a monitoring methodology that collects several different types of complementary data and relies on them to make intelligent decisions about security events. The emphasis is on collecting as much data as is practical, then streamlining an analyst's access to the data to help them get what they need as quickly and easily as possible. For a more detailed explanation, see Chapter 2 of The Tao of Network Security Monitoring. 2.12 Why is it called "sguil"? I won't even begin to go into that, but here's the complete answer. 3.0 Installation and Configuration 3.1 Where do I get sguil? Most people should use the stable released version, available on the Sourceforge download page. Developers or fearless souls can try the CVS snapshot, which is often updated several times a day. You can fetch the latest code with the following command: % cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/sguil checkout sguil Check out the sguil download page for pointers to RPMs, ISOs and other ways to get sguil. 3.2 Should I use the stable version or the CVS development release? CVS usually works, but you might find some lingering bugs from time to time. If you have a healthy tolerance for this sort of thing, try the CVS version. You'll always get the latest code, which often includes significant new features. On the other hand, if you need stability more than bleeding edge functionality, you should definately be using the stable releases. 3.3 What other software does sguil depend upon? The following table summarizes the different tools that make up sguil and shows the different types of information they provide: Tool Purpose URL MySQL 3.x Data storage and retrieval http://www.mysql.com Snort 2.x Intrusion detection alerts, scan detection, TCP/IP session records, packet logging http://www.snort.org Barnyard Conduit between Snort and MySQL http://www.snort.org/dl/barnyard SANCP TCP/IP session records http://www.metre.net/sancp.html Tcpflow Extract an ASCII dump of a given TCP session http://www.circlemud.org/~jelson/software/tcpflow/ p0f Operating system fingerprinting http://lcamtuf.coredump.cx/p0f.shtml tcpdump Extracts individual sessions from packet logs http://www.tcpdump.org 3.4 This is really complicated. Is there an installation and configuration guide? Yes, there are two that I know of. The official sguil installation document is here. Unfortunately, it skips a few steps, like the installation of SANCP and p0f. Richard Bejtlich maintains an installation guide for FreeBSD systems, although I have personally used it to install under Linux with no serious issues. Richard's document is the better of the two, but it's worthwhile to read both anyway. 3.5 How can I tell if it's all working right? There's no standardized test suite, but you can get a pretty good indication by using various client functions. First, make sure you can log in, monitor some sensors and actually see some alerts. That'll show you that the basic sguil database is working and the sguil server is responding to your requests. Next, highlight an alert and turn on the "Show Packet Data" and "Show Rule" buttons. You should see the packet headers and payload displayed in the lower right hand pane, and the Snort rule that generated the alert in the line just above. If you get these, all the basic alerting data is working. Next, you should check to see if you can generate ASCII transcripts from the full packet logs. Select one of the alerts that says where the "Pr" (protocol) column is listed as "6". This is a TCP session, and the only type of session from which you can create ASCII transcripts. Right-click in the "sid.cid" column and select "Transcript" from the context menu. A new window should appear, and eventually it should contain a copy of the payload data from that session. It's worthwhile at this point to also select "Ethereal" from the same context menu, just to make sure that you can not only get the transcript but also the raw packets when you need them. Finally, make sure session reporting is working. Using the same alert as before, right-click in the "Src IP" column and select Query Event Table -> Query SrcIP from the context menu. This generates an SQL query looking for all Snort alerts that come from that same IP address. The client will bring up another window that allows you to modify the SQL query before running it, but in this case you want to run the query as is, so just click "Submit". You may have to wait a few minutes, but eventually you sguil should return at least one event (the event you originally chose), and perhaps others as well. Following the instructions from the last paragraph, repeat this procedure for both the Query Sessions Table -> Query SrcIP and the Query SANCP Table -> Query SrcIP context menu items as well. You should receive similar results. If all these tests worked, your sguil installation is working normally. If any of the tests failed, refer to the Troubleshooting section of this FAQ for pointers, or ask for help on the mailing list or IRC channel. 3.6 How does sguil capture network session information? Sguil uses two distinct methods to capture information about network sessions. The older method is to patch the source code to snort, and enable the stream4 preprocessor directive in your snort.conf file. This allows snort to log information about TCP sessions it sees into the sguil database. The advantage to this method is that it leverages the copy of snort you're already running. This works well, but is limited to logging only TCP sessions. UDP won't be shown at all. The newer method of capturing session data is to use SANCP. SANCP runs in a separate process and logs both TCP and UDP, which gives you a much more complete picture of the traffic on your network. You can run both at the same time if you like, but this can cause significant database performance loss, since it has to do twice as much work to insert both session sources into the database. 3.7 Should I use snort or sancp to capture session information? You are usually much better off running SANCP instead of the snort preprocessor method. For about the same amount of system overhead, SANCP gives you a far better picture of who's talking to who. 4.0 Using Sguil 4.1 How do I set up sguil to automatically categorize incoming alerts? This is called "automatic categorization", or just "autocat" for short. Take a look at /etc/sguild/autocat.conf, which contains full instructions. Once you edit this file, you'll need to restart sguild in order for the changes to take effect. 4.2 Can sguil page me when it sees a particular alert? 5.0 Troubleshooting 5.1 Something's not working. Where do I start? This is a very hard question, since sguil is pretty complex. Probably the best thing to do is to put some of the pieces into debug mode and see what messages you get. Sguild and the sensor_agent are the most likely places to start (see questions 5.2 and 5.3 below). If that doesn't help, you might want to drop by the IRC channel and ask for help. 5.2 Putting sguild into debug mode Locate your sguild.conf file. This is usually /etc/sguild/sguild.conf, but the exact location may vary on your installation. Then edit the line that says "set DEBUG 0" to "set DEBUG 2" (debug level 1 is far less useful for troubleshooting) and restart sguild. While you're editing sguild.conf, you might also want to make sure you have "set DAEMON 0" in the configuration as well. This will prevent sguild from forking into the background when it starts up, which is useful when you're running it in debug mode. If you have "set DAEMON 1" (or if you use the -D option when starting sguild), you'll also get debug output, but it will go into syslog. Note: Versions of sguil prior to 0.5.3 will syslog your database password if DEBUG is 2 and sguil runs in daemon mode. Versions 0.5.3 and newer will replace it with "xxxxx" in the syslog, but will still show it if debug output is going to stdout. 5.3 Putting sensor_agent into debug mode This is just like putting sguild into debug mode, except you edit the sensor_agent conf file on the machine that runs the sensor you are debugging. This is usually /etc/sensor_agent.conf, but this may be different on your installation. Edit the line that says "set DEBUG 0" to read "set DEBUG 1" and restart sensor_agent. At this point, you might also want to make sure you have "set DAEMON 0" in the configuration as well. This will prevent the sensor_agent process from forking into the background when it starts up, which is useful for debugging. If sensor_agent runs in daemon mode and debugging is turned on, the messages will go to syslog. 5.4 I'm seeing IPC problems with sguild, and data isn't being loaded into the database Benjamin Smee ben at disciplina d0t net has this to say: If you are experiencing problems with IPC or seemingly bizare problems with data not being loaded into your DB, thenchances are you have enabled threading in TCL or TCLX. You must DISABLE threading for sguild to work correctly due to the forking nature of how sguild is currently written. Sguild versions 0.5.4 and above have been fixed to exit with a warning message if run inside a threaded Tcl interpreter. If you are using an older version of sguil, you may need to recompile Tcl in order to disable threading. 6.0 Miscellaneous There are no FAQ entries for this section yet. Please contact David if you want to suggest one. Version 0.8, Last Updated 13 December 2004 FAQ Editor: David J. Bianco david at biancorama d0t com Sguil is copyrighted by Bamm Visscher -- Copyright 2006 ©