Logger which parses raw Gatling logs and sends them to the Elasticsearch.

By default, Gatling writes logs to the console, which is inconvenient for analysing, collecting and storing information. In fact, metrics don't contain the details of errors, they only have request status OK or KO. When a metric occurs with an error it's impossible to figure out what happened: a check failed? got 404? or it was 502? etc. Also, if you run load tests in the distributed mode, it will store your logs in separate injectors. This logger allows getting all useful information so that it will be possible to correlate with your metrics.

To recap, the Logger is solving two main problems:

• Distributed metrics sending and storing them
• You can build a Graph with errors details for correlation by metrics

Add to your pom.xml

<dependency>
  <groupId>io.github.amerousful</groupId>
  <artifactId>gatling-elasticsearch-logs</artifactId>
  <version>1.6.1</version>
</dependency>

Add to your build.sbt

libraryDependencies += "io.github.amerousful" % "gatling-elasticsearch-logs" % "1.6.1"

I provide minimal configuration, but you can add additional things what you need

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="ELASTIC" class="ElasticGatlingAppender">
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>${logLevel}</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <url>http://${elkUrl}/_bulk</url>
        <index>gatling-%date{yyyy.MM.dd}</index>
        <type>gatling</type>
        <errorsToStderr>true</errorsToStderr>
        <operation>index</operation>

        <headers>
            <header>
                <name>Content-Type</name>
                <value>application/json</value>
            </header>
        </headers>
    </appender>

    <logger name="io.gatling.http.engine.response" level="${logLevel}"/>
    <logger name="io.gatling.http.action.ws.fsm.WsLogger" level="${logLevel}"/>

    <appender name="ASYNC ELK" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="ELASTIC"/>
    </appender>

    <root level="WARN">
        <appender-ref ref="ASYNC ELK"/>
    </root>

</configuration>

This logger based on https://github.com/agido-malter/logback-elasticsearch-appender which is directly responsible for sending to Elasticsearch. There you can find some addition and useful options related with sending

Pay attention on two variables in config:

elkUrl - URL of your Elasticsearch

logLevel - log's level. DEBUG to log all failed HTTP requests. TRACE to log all HTTP requests

Example how to pass the above variable during the run load test:

mvn gatling:test -DelkUrl=%URL%:%PORT% -DlogLevel=%LEVEL%

Logger can also parse Session attributes and send them to Elasticsearch. As an example, your test might contain some entity id: userId, serverId, etc. It's useful for filtering data. Here is what you need to add to the appender:

<appender name="ELASTIC" class="ElasticGatlingAppender">

    <extractSessionAttributes>userId;serverId</extractSessionAttributes>

</appender>

In my case, I add simulation name to Session. Although metrics writes by simulation name, logs contain only class name, and you can't match them. During several simultaneously running tests, you will get mixed logs which we can filter by simulation name.

Example how to override scenario method and write simulation to Session:

class Example extends Simulation {
  
  override def scenario(name: String): ScenarioBuilder = {
    import io.gatling.commons.util.StringHelper._


    val simulationName = RichString(this.getClass.getSimpleName).clean

    io.gatling.core.Predef.scenario(name)
      .exec(_.set("simulation", simulationName))
  }

  val scn: ScenarioBuilder = scenario("Example scenario")
    .exec(request)
    ...
}

Exclude logs from failed resources in silent mode (NoopStatsProcessor):

 <excludeResources>true</excludeResources>

The principle of works is to parse logs and then separate them by necessary fields. Currently, the Logger supports:

• HTTP
• WebSocket
• SessionHookBuilder

Example of how the Logger parsing a raw log by fields:

>>>>>>>>>>>>>>>>>>>>>>>>>>
Request:
get request: OK
=========================
Session:
Session(Example scenario,1,Map(gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@434d148, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@105cb8b8, gatling.http.cache.baseUrl -> https://httpbin.org, identifier -> ),OK,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$529/0x0000000800604840@1e8ad005,io.netty.channel.nio.NioEventLoop@60723d6a)
=========================
HTTP request:
GET https://httpbin.org/get
headers:
   accept: */*
   host: httpbin.org
=========================
HTTP response:
status:
   200 OK
headers:
   Date: Wed, 25 Aug 2021 08:31:38 GMT
   Content-Type: application/json
   Connection: keep-alive
   Server: gunicorn/19.9.0
   Access-Control-Allow-Origin: *
   Access-Control-Allow-Credentials: true
   content-length: 223

body:
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "X-Amzn-Trace-Id": "Root=1-6125ffea-3e25d40360dd3cc425c1a26f"
  },
  "origin": "2.2.2.2",
  "url": "https://httpbin.org/get"
}

<<<<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>>>>>
Request:
Send Auth: KO Check timeout
=========================
Session:
Session(Example scenario,1,HashMap(gatling.http.cache.wsBaseUrl -> wss://mm-websocket.site.com, ...)
=========================
WebSocket check:
Wait DELETED
=========================
WebSocket request:
{"auth":{"userId":123}}
=========================
WebSocket received messages:
17:50:41.612 [0] -> {"type":"EXPORT_PROCESS","status":"DISPATCHED"}
17:50:41.741 [1] -> {"type":"EXPORT_PROCESS","status":"RECEIVED"}
17:50:41.830 [2] -> {"type":"EXPORT_PROCESS","status":"RECEIVED"}
17:50:42.073 [3] -> {"type":"EXPORT_PROCESS","status":"RECEIVED"}
17:50:44.209 [4] -> {"type":"EXPORT_PROCESS","status":"RECEIVED"}
17:50:44.379 [5] -> {"type":"EXPORT_PROCESS","status":"RECEIVED"}
17:50:44.437 [6] -> {"type":"EXPORT_PROCESS","status":"COMPLETED"}
<<<<<<<<<<<<<<<<<<<<<<<<<

Integration Elasticsearch with Grafana. You can find addition information here

• https://grafana.com/docs/grafana/latest/datasources/elasticsearch/

Example of a graph which based on these logs

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

MIT