newsletter

Ricevi gli ultimi aggiornamenti da Hortonworks tramite e-mail

Una volta al mese, ricevi gli approfondimenti, le tendenze, le informazioni analitiche e la conoscenza approfondita dei big data.

AVAILABLE NEWSLETTERS:

Sign up for the Developers Newsletter

Una volta al mese, ricevi gli approfondimenti, le tendenze, le informazioni analitiche e la conoscenza approfondita dei big data.

invito all'azione

Per iniziare

cloud

Sei pronto per cominciare?

Scarica Sandbox

Come possiamo aiutarti?

* Ho compreso che posso disdire in qualsiasi momento. Sono inoltre a conoscenza delle informazioni aggiuntive presenti nella informativa sulla privacy di Hortonworks.
chiudiPulsante di chiusura
December 18, 2018
diapositiva precedentediapositiva successiva

Monitoring Kafka Streams Microservices with Hortonworks Streams Messaging Manager (SMM)

In last week’s blog Secure and Governed Microservices with HDF/HDP Kafka Streams Support, we walked through how to build microservices with the new Kafka Streams support in HDF 3.3 and HDP 3.1 that is fully integrated with Ranger, Schema Registry and other platform services. This blog is all about monitoring these microservices with Hortonworks Streams Messaging Manager (SMM).

Monitoring MicroServices with SMM

In a microservices architecture, you will see a proliferation of stand-alone decoupled services. Hence, monitoring and managing these services becomes extremely critical. SMM provides users a powerful tool to monitor and visualize their microservices and understand how data flows across these services.

Working off the trucking fleet use case example from the previous blog, you can view each of the three microservices as a consumer and producer as depicted in the below diagram.

Lets first focus on monitoring the stream between MicroService 1 and MicroService 2 where MicroService 1 is a Kafka producer into the driver-violation-events topic and MicroServce 2 is a consumer from that topic. The below video showcases how to monitor the interactions between these two Kafka Streams microservices.

 

As the above video showcased, SMM cured the Kafka blindness for the streams app between the two microservices shedding light on some important information including the rate at which MicroService 1 was producing data, the lag of MicroService 2 and understanding the details of the internal Kafka changelog topics created by the Kafka streams join operator.

Another common use case for monitoring Kafka streams application using SMM is the following:

  • Detect a Kafka Streams microservice that is lagging considerably behind in processing messages.
  • Scale out the Kafka Streams microservice by spinning up new instances of the app (adding new instances to the consumer group).
  • Validate that adding new instances has decreased the lag of the consumer group.

The below video showcases how SMM address this use case.

 

The addition of Kafka Streams to HDF 3.3 and HDP 3.1 integrated with platform services like Ranger, Schema Registry and other platform services provides app developers a comprehensive platform to build secure and governed microservices. With SMM, devops and platform operations teams have enterprise  tools to debug, monitor, and troubleshoot these microservices built using Kafka Streams.

What’s Next

In the next installment of the Kafka Analytics blog series, we walk through the new Hive and Kafka integration for the SQL access pattern.

Lascia una risposta

L'indirizzo email non verrà reso pubblico. I campi obbligatori sono segnalati con *