Machine Data and Big Data

Business applications, Sensors in Factories and Large equipment and their supporting systems are a rich source of data. Every technical component in a functioning system produces logging information.  Business application logs are detailed logs and can contain every transaction the customer or user has executed on that application. It doesn’t matter if the transaction was successful or whether it failed - there are insights you can get from Your Log files. System or maching logs have variety, a large number of data sources and formats which are unique based on the application / system which is generating that. Its fast moving, a typical enterprise creates log files for every second volume of data, data is recorded for events and for time periods. A business transaction will fan out and create 100’s of log and audit table entries.  To get a big picture of your operations environment, this rich and complex technical information needs to be analyzed. You need to be able to get data which is relevant to you and this in turn presents a challenge for the software performance engineer or the Operations team. The Operations team is managing a larger and large number of real and virtual environments and the business is asking for more frequent functionality packed releases into the product. Some typical log files which an enterprise deals with are : • Network packet flow information from routers and switches• Storage subsystem metrics• Database metrics• Application servers from Log4j messages• Microsoft WMI• Web Server information from Apache• Operating system information from AIX, Linux, and VMWare• End User experience.