# kibana Published articles for kibana. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## End-to-end SAP Observability with Elastic, Google Cloud, and Kyndryl: A deep dive DevFeed: [End-to-end SAP Observability with Elastic, Google Cloud, and Kyndryl: A deep dive]() Original publisher: [Read original article]() Author: Valerio Arvizzigno,Francesco Di Stefano Published: 2024-07-08T00:00:00Z Content type: article Language: en Sources: [Elastic Blog - Elasticsearch, Kibana, and ELK Stack]() Topics: [observability](), [Google Cloud Platform (GCP)](), [dashboards](), [log management](), [Monitoring](), [hosting](), [Security](), [networking]() Tags: [anomaly-detection-cloud-migration-sap-monitoring-real-time-analysis-network-visibility-google-c](), [cloud](), [deep-dive](), [elastic](), [enterprise](), [google](), [google-cloud](), [industries](), [infrastructure](), [integrations](), [kibana](), [kyndryl](), [logs](), [metrics](), [monitoring](), [networking](), [observability](), [observability-log-analytics-metrics-business-analytics-cloud](), [operations](), [platforms](), [security]() ### AI overview This article presents a full-stack observability approach for complex SAP environments, developed by Elastic in collaboration with Google Cloud and Kyndryl. It describes using Kibana and Elastic integrations to provide visibility across hybrid infrastructure, applications, business processes, logs, metrics, traces, security, and compliance. ### Source excerpt Companies across almost all industries rely on robust complex SAP systems to power their core operations. At Elastic, with the collaboration of Kyndryl and Google Cloud, we designed a full-stack observability experience for your SAP environment. ## How we improved reporting and monitoring of test automation results DevFeed: [How we improved reporting and monitoring of test automation results]() Original publisher: [Read original article]() Author: Giuseppe Donati Web Test Automation Engineer Not a stereotypical Italian guy; Except Published: 2023-02-15T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [Testing](), [Test automation](), [GitHub Actions](), [Selenium](), [CI/CD](), [Cloud](), [Kafka](), [elasticsearch](), [kibana](), [logstash](), [Grafana](), [Cucumber]() Tags: [article](), [elasticsearch](), [frontend](), [gcp](), [github-actions](), [grafana](), [kafka](), [kibana](), [logstash](), [monitoring](), [quality-assurance](), [selenium](), [test-automation]() ### AI overview The article describes trivago's updated system for executing, reporting, and monitoring Selenium-based end-to-end tests. It covers GitHub Actions workflows, custom runners on Google Cloud, cloud report storage, Kafka and Logstash processing, Elasticsearch and Kibana visualization, and Grafana and Slack alerting. ### Source excerpt Over the last few years, we completely refactored what was described in our previous article about how we use the ELK sta... ## Better URL Search with Elasticsearch DevFeed: [Better URL Search with Elasticsearch]() Original publisher: [Read original article]() Author: Jorge Luis Betancourt Follow Published: 2020-02-11T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [elasticsearch](), [Logging](), [log management](), [tokenization](), [Query (disambiguation)]() Tags: [ascii](), [cardinality](), [elasticsearch](), [kibana](), [logging](), [logs](), [logstash](), [monitoring](), [pipeline](), [protocol](), [tokenization]() ### AI overview This article explains how trivago searches URL components stored in Elasticsearch within an ELK-based logging pipeline. It examines how the Standard Analyzer tokenizes URL-like query strings and discusses preprocessing and flattened-field approaches for matching query-parameter key/value pairs, including their tradeoffs. ### Source excerpt At trivago, we generate a huge amount of logs and we have our own custom setup for shipping logs using mostly ## How to manage nested objects in Elasticsearch documents DevFeed: [How to manage nested objects in Elasticsearch documents]() Original publisher: [Read original article]() Author: Lazarus Lazaridis Published: 2019-05-02T12:30:00Z Content type: tutorial Language: en Sources: [Lazarus Lazaridis]() Topics: [elasticsearch](), [kibana](), [API]() Tags: [elasticsearch](), [how-to](), [kibana](), [nested-objects](), [opensource](), [programming](), [script](), [scripts](), [update]() ### AI overview This tutorial explains how to manage nested objects in Elasticsearch documents using Kibana. It covers defining a nested mapping, indexing a document containing cats, and adding, removing, and updating nested objects through the Update API and scripts. ### Source excerpt In this post we are going to manage nested objects of a document indexed with Elasticsearch. The nested type is a specialised version of the object datatype that allows arrays of objects to be indexed in a way that they can be queried independently of each other. - Nested datatype - Official Elasticsearch reference Prerequisites To follow this post you need: an up and running Elasticsearch instance I use 6.7 here an up and running Kibana instance to interact with Elasticsearch Preparation The document of our index will represent a human and its nested objects will be cats (no surprises). Create the index Open your Kibana dev console and type the following to create the index. # Create the index PUT iridakos_nested_objects { "mappings": { "human": { "properties": { "name": { "type": "text" }, "cats": { "type": "nested", "properties": { "colors": { "type": "integer" }, "name": { "type": "text" }, "breed": { "type": "text" } } } } } } } Human has: a name property of type text a cats property of type nested Each cat has: a colors property of type integer a name property of type text a breed property of type text Add a human In the Kibana console, execute the following to add a human with three cats. # Index a human POST iridakos_nested_objects/human/1 { "name": "iridakos", "cats": [ { "colors": 1, "name": "Irida", "breed": "European Shorthair" }, { "colors": 2, "name": "Phoebe", "breed": "European" }, { "colors": 3, "name": "Nino", "breed": "Aegean" } ] } Confirm the insertion with: GET iridakos_nested_objects/human/1 You should see something like this: { "_index": "iridakos_nested_objects", "_type": "human", "_id": "1", "_version": 1, "found": true, "_source": { "name": "iridakos", "cats": [ { "colors": 1, "name": "Irida", "breed": "European Shorthair" }, { "colors": 2, "name": "Phoebe", "breed": "European" }, { "colors": 3, "name": "Nino", "breed": "Aegean" } ] } } Done, moving on. Managing nested objects Add a new nested object Suppose that iridakos got a new Persian ## SAML SSO Authentication for Splunk with G Suite DevFeed: [SAML SSO Authentication for Splunk with G Suite]() Original publisher: [Read original article]() Author: kishlaya kumar Published: 2018-04-12T07:00:30Z Content type: tutorial Language: en Sources: [CleverTap]() Topics: [saml](), [Single sign-on (SSO)](), [Authentication](), [Google](), [log management](), [elasticsearch](), [kibana](), [logstash]() Tags: [analytics](), [authentication](), [authorization](), [browser](), [elasticsearch](), [elk](), [engineering](), [google](), [kibana](), [log-management](), [logstash](), [saml](), [security](), [sso](), [technology](), [technology-engineering]() ### AI overview A step-by-step guide to configuring SAML-based single sign-on for Splunk using Google (G Suite) as the identity provider. It explains the SAML authentication model and flow, following the authors' move from ELK to Splunk for on-premises log management and analytics. ### Source excerpt From early on, our team used ELK (Elasticsearch-Logstash-Kibana) for log management and analytics. ELK served us well, but as our The post SAML SSO Authentication for Splunk with G Suite first appeared on CleverTap. ## ElasticSearch 最佳实践 DevFeed: [ElasticSearch 最佳实践]() Original publisher: [Read original article]() Author: Joway Published: 2017-05-28T00:00:00Z Content type: tutorial Language: zh Sources: [Random Thoughts]() Topics: [elasticsearch](), [Amazon Web Services](), [kibana](), [Ansible](), [Kafka]() Tags: [ansible](), [aws](), [cpu](), [elasticsearch](), [kafka](), [kibana](), [ssd](), [tech]() ### AI overview A practical guide to tuning and operating Elasticsearch clusters, covering hardware and JVM memory settings, CPU behavior during queries and force merges, swap, node and cluster configuration, shard recovery, rolling restarts, and logging pipelines. It also discusses using Kafka as a buffer and Ansible for consistent cluster management. ### Source excerpt Elasticsearch 是一个需要不停调参数的庞然大物 , 从其自身的设置到JVM层面, 有着无数的参数需要根据业务的变化进行调整。最近采用3台 AWS r3.2xlarge , 32GB, 4核, 构建了一套日均日志量过亿的 EFK 套件。经过不停地查阅文档进行调整优化 , 目前日常CPU占用只在30% , 大部分 Kibana 内的查询都能在 5s ~ 15s 内完成。 ## Distributed Troubleshooting DevFeed: [Distributed Troubleshooting]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-04-05T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [big-data](), [incident](), [SIEM, Security, Observability](), [systems](), [elasticsearch](), [Open Source](), [telemetry]() Tags: [big-data](), [data](), [distributed](), [elasticsearch](), [elk](), [incident](), [infrastructure](), [kibana](), [logs](), [logstash](), [metrics](), [open-source](), [open-source-components](), [troubleshooting]() ### AI overview The article describes Target's Distributed Troubleshooting Platform for investigating issues across a large open source big data platform. It aggregates service logs and metrics so engineers can inspect information from many machines and services in one place, using open-source components and the Elasticsearch, Logstash, and Kibana stack. ### Source excerpt Target's open source big data platform contains a vast array of clustered technologies or ecosystems working together. Troubleshooting an issue within a single ecosystem is a difficult task let alone an issue that spans several ecosystems. It is impractical for a single human to individually investigate ecosystems one at a time for potential problems. The house will burn to the ground long before an engineer can find the cause of an issue and resolve it without quick access to aggregated system metrics and logs. The Solution How to identify, troubleshoot and resolve a distributed issue? Fight fire with fire of course! Big data issues must be solved with big data solutions. At Target, we are constantly expanding our Distributed Troubleshooting Platform to encapsulate every log and metric from every service in every ecosystem of our big data platform. Aggregating this data into a single troubleshooting platform enables an engineer to view error logs and system metrics across hundreds of machines and services with a single click. A troubleshooting platform like the one described above is not a new idea. Systems like Splunk have been doing it for years. Splunk however, has restrictions on the amount of data that can be ingested without an enterprise license. The larger we scale; the more money we pay for systems like Splunk. We created our Distributed Troubleshooting Platform from open-source components and without enterprise licenses. This allows us to utilize it on every server in the big data platform without worrying about the volume of data it is processing and re-negotiating enterprise licenses. It becomes a given, not a variable. Our Distributed Troubleshooting Platform is similar to the black box recorder on an aircraft. A majority of the time, the contents are never viewed. When the plane crashes however, the contents of the black box are the only way to reconstruct what happened and learn from the incident. Running a big data platform without enterprise licens ## Go and structured logging with ElasticSearch DevFeed: [Go and structured logging with ElasticSearch]() Original publisher: [Read original article]() Author: sasha@goteleport.com (Sasha Klizhentas) Published: 2016-01-01T00:00:00Z Content type: tutorial Language: en Sources: [Teleport]() Topics: [Logging](), [log management](), [Go Language](), [elasticsearch](), [kibana](), [Docker]() Tags: [beats](), [docker](), [elasticsearch](), [format](), [framework](), [go](), [kibana](), [logging](), [udp]() ### AI overview This tutorial describes an experiment that uses Go, logrus, and Elastic Beats to emit structured logs over UDP and ship them to Elasticsearch. It also covers running Elasticsearch and Kibana with Docker and configuring the document schema and mappings. ### Source excerpt We are playing with Elastic Beats, doing structured logging with Golang and Elastic Search ## Elasticsearch and Kibana for Selenium Automation DevFeed: [Elasticsearch and Kibana for Selenium Automation]() Original publisher: [Read original article]() Author: Teodor Rupi Follow Published: 2015-12-02T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [kibana](), [Selenium](), [Testing](), [Automation](), [Data visualization](), [elasticsearch](), [Jenkins](), [Kafka](), [logstash](), [ci](), [Maven]() Tags: [apache](), [automation](), [backend](), [chrome](), [ci](), [data-visualization](), [devops](), [elasticsearch](), [firefox](), [java](), [jenkins](), [kafka](), [kibana](), [linux](), [logging](), [logstash](), [mac](), [mac-os](), [maven](), [monitoring](), [selenium](), [testing]() ### AI overview This article describes trivago's Selenium-based automated testing infrastructure and its use of Kibana for real-time reporting, filtering, and analysis of test results. It outlines a setup that uses Jenkins, Kafka, Logstash, Elasticsearch, and Kibana, with testing across multiple platforms and browsers. ### Source excerpt The advances and growth of our Selenium based automated testing infrastructure generated an unexpected number of test results to evaluate. We had to rethink our reporting systems. Combining the power of Selenium with Kibana's graphing and filtering features totally changed our way of working. ## Writing a Fuzzy Receipt Parser in Python DevFeed: [Writing a Fuzzy Receipt Parser in Python]() Original publisher: [Read original article]() Author: Matthias Endler Published: 2015-10-06T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [Python](), [Parser](), [Hackathon](), [CSV](), [elasticsearch](), [kibana]() Tags: [csv](), [elasticsearch](), [engineering-culture](), [hackathon](), [kibana](), [open-source](), [python](), [scanner](), [usb]() ### AI overview A Python hackathon project explores building a fuzzy receipt parser for household expense tracking. The tool is intended to scan receipts, identify the shop, date, and total with more than 90% precision, and export the results to CSV for later analysis. ### Source excerpt Last weekend, the Python Hackathon Düsseldorf took place at trivago's office. Although we were only five people we had a lot of fun. I took the chance to brush up my Python skills a little bit. Also I wanted to scratch an itch that was bugging me for a long time: our housekeeping book. ## Fixing Discourse performance regressions DevFeed: [Fixing Discourse performance regressions]() Original publisher: [Read original article]() Author: Sam Saffron Published: 2015-10-02T07:26:25Z Content type: tutorial Language: en Sources: [Sam Saffron]() Topics: [Development](), [Rails](), [debugging](), [Grafana](), [Tooling]() Tags: [debugging](), [grafana](), [kibana](), [performance](), [performance-tuning](), [rails](), [sql]() ### AI overview A Discourse performance regression is used to explain a methodology for performance debugging and optimization. The article recommends prioritizing heavily used foreground routes, monitoring traffic and long-term performance trends, establishing baselines and goals, and using tools such as Kibana, Grafana, rack-mini-profiler, and flamegraphs. It also warns that Rails performance tuning in development mode can produce misleading noise compared with production. ### Source excerpt Recently, I discovered a performance regression on a very common page on Discourse. I spent a fair amount of time debugging and optimizing. I follow a certain methodology while I do this kind of work. This post is a breakdown on the specific issue I faced with some points you can take back and apply to your next performance debugging session. Pick your fights The first and most important point to take is that you should pick your battles. Discourse has hundreds of routes, however the vast majority of the server cost is incurred by a handful. Pasted image792x576 22 KB The most important 3 routes for us are "topics/show", "list/latest" and "categories/index". They are the heart of the site and lion's share of foreground routes. "topic/timings", "user avatars" and "drafts" are all background routes, we still want to minimize work on them so servers work less hard and we can host more sites, however slowness there is usually not observed by end users. I always try to focus first on the most active foreground routes, those are the spots where I will invest the most amount of effort optimizing. To get a good picture of our traffic patterns we use Kibana. To answer the same question you may use Google Analytics, New Relic or some other tool. Start with a baseline and a goal We have a Grafana dashboard keeping an eye on our 2 most important routes for every site we run. I visit the dashboard regularly to see how performance is on those routes. Is displaying topics getting faster or slower? It is very important to have long term trends so you can isolate when stuff starts playing up. Recently I discovered this: Pasted image971x271 56.7 KB Showing topics on 2 particular sites (one is shown) got much slower. Having this information is golden. This graph is a visible report card on my work towards improving performance. When I see a graph like this my immediate goal becomes restoring old performance characteristics. This is particularly important here since this is our most imp ## 10x: Logging at Clay.io DevFeed: [10x: Logging at Clay.io]() Original publisher: [Read original article]() Author: Zoli Kahan Published: 2014-10-25T05:00:00Z Content type: article Language: en Sources: [Zolmeister]() Topics: [Logging](), [log management](), [logstash](), [elasticsearch](), [kibana](), [Amazon VPC](), [Server](), [Linux](), [Docker]() Tags: [aggregate](), [amazon](), [amazon-vpc](), [analyze](), [apply](), [architecture](), [complex](), [docker](), [elasticsearch](), [kibana](), [linux](), [logging](), [logs](), [logstash](), [network](), [series](), [server](), [servers](), [ssh]() ### AI overview This article describes how Clay.io used Logstash to aggregate logs from more than 20 servers, with Elasticsearch and Kibana for analysis. It also discusses log rotation, securing Elasticsearch through Amazon VPC, and open-sourced Docker containers for deploying a distributed logging system. ### Source excerpt 10x: Logging at Clay.io Managing 20+ servers as a small team is no easy task, and when things go wrong (they always do) figuring out what happened quickly is essential. Of course we can't ssh into each machine, that would take ages, so instead we use Logstash to aggregate our logs. This is the second post in my series, and if you missed last episode: Architecture at Clay.io. Logstash overview Logstash deployments have two parts. The aggregate server (or cluster), and the client servers.