Our process of adding data to ClearSite differs from the way Cacti/mac-track does on a few different levels:
Add a location: At least one location must be defined to use ClearSite, and this is taken care of during the setup script. Additional locations (city, state, country, address, contacts, timezone...) can easily be added or deleted at any time.
Adding devices: Devices can be added using a simple TSV (tab seperated value) document for quick imports, or devices can be added one by one in the Admin interface.
Once devices are entered into ClearSite, the db_walk and rrd_walk cron jobs get kicked off.
Db_walk: The db_walk script is responsible for gathering the data from the devices that have been entered into the admin interface, and then placing that data into the database. The data gathered consists of such things like the names, descriptions, status, ip's, mac-address, owner, service... etc... basically anything that is not a counter or guage on the device being walked.
Rrd_walk: The rrd_walk relies on the db_walk script to write to the database about various interfaces, owners, services, cpu's etc... rrd_walk checks the database for interfaces with an operational status of "up", and then walks all the counters for any up interfaces or other specified entity such as owner or content rule. Rules/Services/Interfaces/Ports that are marked as down, do not get walked. Also, currently, there is no formal way in ClearSite to ignore an interface, cpu, port etc... If it's up, it's walked.
That's it in a nut shell, below I am going to list all the other neat stuff db_walk takes into account and why: Cisco switches are kooky to say the least, and gathering data via snmp is great, however there are quite a few nuisances they have that... well... seem counter intuitive to put it politely.
Trunks and CDP ports:
Trunked ports provide a quandary, because most switches that connect to other switches, espically ones with more than one vlan, typically do so over trunked ports. Cisco switches share the mac address tables with one another and keep track of where they learned those mac's from. Since Cisco switches keep track of one anothers MAC address tables, there are many many more MAC address' listed in the SNMP tables than there (physically) actually are on the device itself. This would be an easy problem if other ports didn't use trunks, but many do, take a vmware server for example, if that server houses other virtual servers on different subnets/vlans then its very likely the port is trunked. ClearSite works around the issue by looking for CDP neighbors, CDP is the cisco discovery protocol that most cisco network equipment uses to find out about other cisco equipment. ClearSite looks at the CDP tables, and ignores all mac address' learned on CDP ports. This way trunked ports, like in the VmWare example above, don't have to be ignored.
Grouping and OperStatus:
ClearSite also groups ports on switches by blade automatically. For example, a 6509 might have five (48 port) blades in it, and looking at a list of 240 could take a long time to scroll through. So we decided to have ClearSite group the ports into the blades, and all one has to do is expand that blade to see those 48 ports. In addition to grouping, ClearSite only gathers data for ports that have a status of "up", typically meaning connected. Error disabled, or interfaces that are admin down are still walked if the connection is still there. The operational status also narrows down the total list of ports on most blades, so if only 5 ports are connected, you only see a list of 5 when you expand the blade, not all 48 ports. If a port becomes active again or for the first time, it's added to the list and the rrdfile is then created, unless it already existed, then the data is simply written to the rrd file again.
Storing Data:
We decided to store more data in the ClearSite default rrd files than Cacti, which does result in lager file sizes for the rrd's, but we feel it's better this way. Each traffic.rrd file is about 1.7 megs, and this allows to store the in/out counters for each interface in more detail. When you look at today's data for interface-x, and when you go back a week from now to look at that same time frame, the graphs will be identical for that interface. This also aids in getting a clearer picture of the data over longer periods of time without the "consolidation loss" you can get with Cacti's
default rrd's. In addition to the rrd data, we keep track of the IP's that MAC address' have been assigned to, to better track who was doing what on which day. This also helps from a security aspect, if for example you needed to know the name of the pc for this date, or even the time that pc's IP changed, you will know.
Normalized Data:We thought long and hard about how all the pieces of the database should fit together, and we feel we've done a fine job of it. We attempted to cut out repetitive data from the db as well as using foreign keys to help maintain the data's accuracy. While were not DBA's, and just have a few PHP/MySQL books to draw from, we feel like we accomplished our goals of normalization.
Naming Conventions:In keeping with our normalization ideas, we decided to make the rrd file names follow some sort of repeatable standard, and not just simply auto-increment the file names, and store their associations in the DB, but instead make the file names meaningful. The convention we came up with is very simple... For example, fast-ethernet port 1/1 on a router with an ip of 1.2.3.4 would appear as: 1.2.3.4.fa1-1.trrafic.rrd
or a subinterface (number 5) on the same port:
1.2.3.4.fa1-1.5.traffic.rrdA gigabit ethernet port on a switch could be:
2.3.4.5.gi1-1.traffic.rrdCpu's are like the following:
2.3.4.5.0.cpu.rrdThis lessens the amount of calls to the DB and speeds up access to the rrd's, we feel.
AJAX ScriptsLast but certainly not least is the use of AJAX in our search, administration, and navigation tree. This is one of the best features of our offering, and we'd like to take credit for it, but we actually grabbed the scripts from
TwinHelix.com. They work very well in all browsers, and they are super easy to use. Overall, we are very happy with the progress we've been able to make.
More to come!!We plan to have much more over the next few months, we've learned a lot from other great and wonderful tools like Cacti, NetDisco, Nagios and many many more. We feel we are standing on the shoulders of giants, and can only dream that others will accept and give back to our project like they have with those other superb tools.
-rich
