ClearSite Blog

ClearSite Beta 4.20

2008-08-28T10:02:00.000-07:00

Clearsite Latest Release August 26th 2008!

We are proud to announce the latest beta version 4.20! This is the 3rd beta release, and final beta, going forward we will produce Release Candidates, and finally the Production Ready Releases. Visit the Clearsite Sourceforge.net Download page, and leave your feedback, feature requests and bugs on the mailing lists.

ClearSite is a Free Open-Source Network Graphing and Management System and much more. ClearSite takes advantage of the powerful RRDTool storage and graphing tool, as well as utilizing MySQL for the backend. We have spent just as much time working on the user experience as we have with the data collection, we know you'll find ClearSite easy to use and understand.

ClearSite has been developed and tested extensively with Cisco networking equipment, as well various PC/Server manufacturers and Operating Systems. *The current release is for Linux only*

Upcoming features for Clearsite Release Candidates

ClearSite RC1 will be a partial rewrite of functions to allow user contribution, easy customization, and granular device control. The existing database structure will remain the same with some added tables and fields.

Windows Install: Windows will be supported, and recommended for Active Directory WAN/LAN environments.
Nagios / SmokePing like Functionality: Utilizing Fping, we are developing Nagios/SmokePing like host checking, latency alert system. We are also looking at an acknowledgment system that accepts input from multiple sources such as SMS message, IM, Email, and naturally the web frontend.
Thresholding and Alerts:: Users will be able to create their own alerts and thresholds for hosts/ports/interfaces they are interested in.
More than just Cisco: More than just Cisco gear will be supported, especially for MAC-to-IP correlation for more accurate search results
Easy Templating and Script Integration: We have designed a XML template that allows extreme flexibility and ease of use and will aid greatly in user contributions and customization
Snort DB Integration: ClearSite will be able to look through your Snort DB and gather all information automatically about the host that triggered an alert. Such as logged on username, PC data, running process's, and naturally the graph data for the switchport and host.
Complete Asset Control: Track host's by HW Serial, UUID/GUID, Mac address, hostname, SID etc... In addition to custom asset tags. All of which can be searched for using our wildly popular AJAX interface!
Universal Outage/Postmortem Database: Each group will have their own custom outage report and postmortem. Alerts, follow ups and postmortems will all be managed and maintained in the ClearSite outage database. Reports will be easy to locate and even update from the web interface, additionally reports can be output into the following formats: Plain-Text, HTML, PDF, and CSV.

ClearSite Beta Released to the Public!

2008-07-01T06:17:00.000-07:00

The procrastination is over, we are offering our first public Beta as of July 1st 2008! We've been inundated with emails and we are happy we can now tell them where they can go... That's right, all of you can go to SourceForge.net!
Naturally we'd like to polish off all the little features and gadgets we want to provide, but we'd have to push the release back again if we did that. So without further ado, we are very proud and relieved to offer the ClearSite Network Management System!

A Long Overdue Enterprise Network Management Sytem (Aloe NMS for short) awaits anyone who wants a NMS that does the work for them, rather than doing the work for the NMS. While other tools are similar, we feel the user and especially administration experience is better than anything else! Please feel free to contact us with your questions and comments!

We will be turning up our own forums, support and documentation webs sites very soon!
http://aloeusers.net (User support, Documentation and user contibutions!)
http://sourceforge.net/projects/clearsite/ (SourceForge.net)
http://aloenms.net (Latest news and updates)
http://dev.clear-site.net (CVS and developement releases)
http://clear-site.info (OfficalWiki and Documentation)
http://xinn.org/ClearSite.html

We'd like to thank everyone who contributed and supported us:
Chris Brodnik (Lead developer and code monkey)
Rich Rumble (Project Manager and Designer | Everything else ;)
EthosPrime (Beta Testers and feature requests;)
Xinn.org Clientele (Beta Testers and feature requests, thanks for adding to our scope creep!)
Angus Turnbull - For allowing us to package and redistribute his awesome AJAX scripts!

Day Jobs and Delays

2008-03-05T07:38:00.000-08:00

There are only two of us coding and working on ClearSite, and we both have day jobs that are keeping us from finishing our first release. We are working on the install script, requirements and documentation. We must apologize for the delay, and every time I look at the analytics for our project and websites, I get more and more anxious to finish the release. The good news is, one of us is loosing their job soon, so there should be more free time to work on this. Please stay tuned!
-rich

Our Process

2007-12-28T10:51:00.000-08:00

Our process of adding data to ClearSite differs from the way Cacti/mac-track does on a few different levels:
Add a location: At least one location must be defined to use ClearSite, and this is taken care of during the setup script. Additional locations (city, state, country, address, contacts, timezone...) can easily be added or deleted at any time.
Adding devices: Devices can be added using a simple TSV (tab seperated value) document for quick imports, or devices can be added one by one in the Admin interface.

Once devices are entered into ClearSite, the db_walk and rrd_walk cron jobs get kicked off.

Db_walk: The db_walk script is responsible for gathering the data from the devices that have been entered into the admin interface, and then placing that data into the database. The data gathered consists of such things like the names, descriptions, status, ip's, mac-address, owner, service... etc... basically anything that is not a counter or guage on the device being walked.

Rrd_walk: The rrd_walk relies on the db_walk script to write to the database about various interfaces, owners, services, cpu's etc... rrd_walk checks the database for interfaces with an operational status of "up", and then walks all the counters for any up interfaces or other specified entity such as owner or content rule. Rules/Services/Interfaces/Ports that are marked as down, do not get walked. Also, currently, there is no formal way in ClearSite to ignore an interface, cpu, port etc... If it's up, it's walked.

That's it in a nut shell, below I am going to list all the other neat stuff db_walk takes into account and why: Cisco switches are kooky to say the least, and gathering data via snmp is great, however there are quite a few nuisances they have that... well... seem counter intuitive to put it politely.
Trunks and CDP ports:
Trunked ports provide a quandary, because most switches that connect to other switches, espically ones with more than one vlan, typically do so over trunked ports. Cisco switches share the mac address tables with one another and keep track of where they learned those mac's from. Since Cisco switches keep track of one anothers MAC address tables, there are many many more MAC address' listed in the SNMP tables than there (physically) actually are on the device itself. This would be an easy problem if other ports didn't use trunks, but many do, take a vmware server for example, if that server houses other virtual servers on different subnets/vlans then its very likely the port is trunked. ClearSite works around the issue by looking for CDP neighbors, CDP is the cisco discovery protocol that most cisco network equipment uses to find out about other cisco equipment. ClearSite looks at the CDP tables, and ignores all mac address' learned on CDP ports. This way trunked ports, like in the VmWare example above, don't have to be ignored.
Grouping and OperStatus:
ClearSite also groups ports on switches by blade automatically. For example, a 6509 might have five (48 port) blades in it, and looking at a list of 240 could take a long time to scroll through. So we decided to have ClearSite group the ports into the blades, and all one has to do is expand that blade to see those 48 ports. In addition to grouping, ClearSite only gathers data for ports that have a status of "up", typically meaning connected. Error disabled, or interfaces that are admin down are still walked if the connection is still there. The operational status also narrows down the total list of ports on most blades, so if only 5 ports are connected, you only see a list of 5 when you expand the blade, not all 48 ports. If a port becomes active again or for the first time, it's added to the list and the rrdfile is then created, unless it already existed, then the data is simply written to the rrd file again.
Storing Data:
We decided to store more data in the ClearSite default rrd files than Cacti, which does result in lager file sizes for the rrd's, but we feel it's better this way. Each traffic.rrd file is about 1.7 megs, and this allows to store the in/out counters for each interface in more detail. When you look at today's data for interface-x, and when you go back a week from now to look at that same time frame, the graphs will be identical for that interface. This also aids in getting a clearer picture of the data over longer periods of time without the "consolidation loss" you can get with Cacti's default rrd's. In addition to the rrd data, we keep track of the IP's that MAC address' have been assigned to, to better track who was doing what on which day. This also helps from a security aspect, if for example you needed to know the name of the pc for this date, or even the time that pc's IP changed, you will know.
Normalized Data:
We thought long and hard about how all the pieces of the database should fit together, and we feel we've done a fine job of it. We attempted to cut out repetitive data from the db as well as using foreign keys to help maintain the data's accuracy. While were not DBA's, and just have a few PHP/MySQL books to draw from, we feel like we accomplished our goals of normalization.
Naming Conventions:
In keeping with our normalization ideas, we decided to make the rrd file names follow some sort of repeatable standard, and not just simply auto-increment the file names, and store their associations in the DB, but instead make the file names meaningful. The convention we came up with is very simple... For example, fast-ethernet port 1/1 on a router with an ip of 1.2.3.4 would appear as: 1.2.3.4.fa1-1.trrafic.rrd
or a subinterface (number 5) on the same port: 1.2.3.4.fa1-1.5.traffic.rrd
A gigabit ethernet port on a switch could be: 2.3.4.5.gi1-1.traffic.rrd
Cpu's are like the following: 2.3.4.5.0.cpu.rrd
This lessens the amount of calls to the DB and speeds up access to the rrd's, we feel.
AJAX Scripts
Last but certainly not least is the use of AJAX in our search, administration, and navigation tree. This is one of the best features of our offering, and we'd like to take credit for it, but we actually grabbed the scripts from TwinHelix.com. They work very well in all browsers, and they are super easy to use. Overall, we are very happy with the progress we've been able to make.
More to come!!
We plan to have much more over the next few months, we've learned a lot from other great and wonderful tools like Cacti, NetDisco, Nagios and many many more. We feel we are standing on the shoulders of giants, and can only dream that others will accept and give back to our project like they have with those other superb tools.
-rich

Asinine Applications

2007-11-23T14:30:00.000-08:00

Why does your company use that software, that application that everyone dreads? Why are the people who bought/told you the software to use, not the people who will use the software themselves? Why is the software that was demo’d to you and your team, not the software you ended up buying? Why does the application your using require you to have it customized, adding to the cost of the software again?

I’ve seen this at every company I’ve worked for in one form or another. It seems to have gotten worse over the years. Having software that no one likes, and continues to use day after day that is OK as long as you’re (honestly) looking to replace that software soon. Perhaps you just needed something quickly, and hopefully you had to decide between the lesser of two evils. But if you have folks using software they can’t stand day in and day out, and you never actually look for better or more focused software, you or the people choosing the software for you, are hurting your company. What’s worse, maybe you chose software no one liked, for whatever reason, and now you can feel the hatred multiply with each login to that app.

Typically it’s not that bad, it’s almost to be expected in some situations. But what if you use four or more systems/applications that seem counter intuitive or repetitive? Sometimes notifications and alerts don’t need the same level of detail for a manager as a programmer needs. Some details and data needs to be redacted and or summarized into layman’s terms from time to time.

The company I’ve worked for the last 8+ years seems to be making worse and worse software choices. With each “changing of the guard”, things get worse before they get better. New management comes in, and typically brings with them the people who they already trust from their former company. In doing so, they all bring with them the “this is how we did things at xyz corp” and that may have worked for xyz... This is then followed by spending money. Somehow spending money seems to be better than saving money. I can understand spending wisely, but spending for the sake of spending and being able to say “Well, we bought “this-n-that”, as well as “that-n-this”, which should be all we need”, it’s just asinine.

After talking to friends and family, and it’s the same song all over, across all sections of the workforce, and not just IT or Development. Then I talked to a friend at Google, and they don’t use off the shelf anything, and if they do, they change/mold it to suit as soon as they get it. Apps that don’t work or no one likes, typically get fixed and or replaced very quickly. Google seems to be one in a million, and they seem to be using logic… Well nuts to that!

Google takes risks, and the upper echelons at my current job do not. The only risk in their mind is asking for money, that’s the risk for them. They aren’t asking their users for input, they aren’t drawing on the knowledge of their employees’ talents or experience. Most of them must think that asking for input is weak, shows a lack of knowledge or leadership… Not asking for your users/employees input is asinine, as is picking the biggest "name-brand" vendors software because... well I have no idea. To me it seems vendors and big-name brands aren't for the common man. Sure there are new things like SOX/HIPPA/PCI to think about, but their software doesn't bring you more than one step closer to compliance with any of those.

That’s why I came up with ClearSite. There were apps out there that do (most of) what ClearSite does, but not the way I liked or thought was as useful. So I’ve directed and written as much of ClearSite as I can. I feel it’s the best product on the market, and it’s also one my current employer will never use because it’s free. It’s open-source, we offer support that they can pay for… so who knows, maybe they will. Doubtful, their money is spent on an application that has taken months to get working and doesn't even give you human readable error messages. I could break ClearSite, charge a whole lot of money for it, or charge a license fee for each device that is walked... then they would buy it for sure! I'll really have to break it good...

-rich

Statistics, Bikinis and Aloe

2007-10-12T18:05:00.000-07:00

Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital. ~Aaron Levenstein

Hello and welcome to the ClearSite blog! In this blog I am chronicling the fruition of A Long Overdue Endeavor, ALOE for short. Lets see, how should this story begin... well I'm no writer and have a GED, so let's go all over the map!

I've worked at the same large ".com" for the last 8+ years, and in that time I've come to see how and why most bad decisions are made. I've dreamed of an application like ClearSite for the last 7 years. It started out simple, we created an inventory system with perl, batch files and logon scripts. The logon script ran a batch file, that happened to call an exe called Machine.exe from a PC game, it retrieved all the vital information rather quickly, CPU speed, mem free/used/total, same with the disk. It then copied the file to a share that got parsed with perl which output a web page. We could see what users were about to run out of space, which was a bigger problem back then, and even the last date they logged in, among other vital info(virus def's, OS, SP level etc...).

We had a lot of turn over in the sales area, still do, and it was hard to keep up with where the new users were. I thought it'd be neat to have a floor plan on a web page you could mouse over and see the user who was located there. Well after a lot of research we found out how we could do it, and sure enough we made the page! It was nice and neat we thought at any rate. It stood p for a few months and was rolled out to our biggest office. Everything was fine and dandy. But for some reason, which was never made clear, we were told to stop using it in favor of HP-OpenView and NetIQ. We had nagios (formerly netsaint) running, still do, and we later rolled out Cacti, both major improvements over HPOV and NIQ. We've had many other management suites since, and what seems like dozens of HelpDesk applications, all suck except Cacti and Nagios.

Cacti is great! But over the years, it's become hard to keep up with, and I thought it could be better in a few areas. So rather than contribute code to cacti, I decided to recreate similar functions. Mostly because of the learning experience, and just to see if we could do as good as they have. We are just about there... we've gone through three totally different code bases, mostly trying to be as efficient as we could. This time I think were working on the final version(3) and expect to release it on sourceforge shortly as a beta, even though we've been using one version or another since April o7.

That's it for now, more about ClearSite/ALOE next time!
-rich