Unlike Apache Solr, Elasticsearch has few components that are separate from the search engine. Elasticsearch is search engine; however, it does not have user friendly GUI to configure or view indexes as in the Apache Solr.

In Elasticsearch world, Kibana is visualization component that is installed separately from Elasticsearch engine. Logstash component parses and indexes log files into Elasticsearch engine.

What are the costs?

1. The cost to you is one junior system engineer for half a day.
2. Hardware for hosting Elasticsearch and Kibana if you do not like to install in your existing hardware. These two are harmless applications and can be hosted in your existing hardware that has room.
3. If you would like to get more matrix and more processing of logs, then you will need to customize logstash parsing. This cost depends on your need of how you would like to see the logs. This is more of gravy than need.
4. Maintenance cost is non-existing to low compare to the benefit you are getting out of it.

What are the benefits?

1. Centralized Logging that allows to have all application logs under one roof
2. Search engine provides a capability to search across many different logs
3. Kibana provides a tool to visualize search results in chart as well as text form without any additional cost
4. logstash parses and indexes Apache logging including Log4j out of the box. The tool provides many different hooks to customize the parsing and indexing.

How do you secure?

1. Securing Data
   When this is deployed inside a firewall within WAN, then it is inside an enterprise such that the logs may not be secured as opposed to when you are transferring data across network within inter-enterprise. If there is a need for data security, then we must use HTTPS for both indexing the data and also for accessing the data through Kibana.

This article is about walk through of how you can setup and process logs that are indexed and searchable in 1 hour.

1. Download Elasticsearch components from (www.elastic.co)
   1. Elasticsearch – a Search Engine to index the log files for quick search and visualization
   2. Kibana  – Data Visualization Component for Elastic Search. Unlike Apache-Solr, Elastic Search has separated the visualization component from the Search Engine. This is a good decision; however, it adds an extra step in deploying and maintaining the system.
   3. Logstash – Parser and Indexer for logs. This component is crucial in making the whole log parsing and searching work with less effort. Logstash works out-of-box for Log4J logging. It exposes control on how a log file is processed through configuration files. Therefore, one could write a really good regular expression to slice through the log so that the Logstash can parse and index. Alternatively, one could extend the Logstash to customize the index and thus the visualization.
2. Install those component. This is basically unzipping and copying into a directory of choice (both for Linux and Windows)
3. Start Each component
   1. Elasticsearch –

    bin/elasticsearch

   1. Kibana

    bin/kibana

   the url : - http://localhost:5601

   1. Logstash – before you start the log stash you need a configuration file which has definition of log location and Elasticsearch engine location.

   For simple test:

    logstash -e 'input { stdin { } } output { stdout {} }'

    bin/logstash agent -f example-file.config

4. 1. Logstash configuration for parsing and indexing log files

   input {
   file {
            type => "trace"
       path => [ "/pathA/*trace.log", "/pathB/*trace.log", 
                  "/pathA/workflow.log", "/pathB/workflow.log"]
       start_position => "beginning"
     } 
    
   } 
    
   filter {
     multiline {
       type => "trace"
       pattern => "(?!header\b)\b\w+"
       what => "previous"
     }
   }
    
   output {
     stdout { codec => rubydebug }
     elasticsearch {
       host => localhost
       port => "9200"
       protocol => "http"
     }
   }
    
   

Commercial Solutions

There are commercial solutions that does similar solutions. Splunk is one of such solutions.
Splunk splunk