Welcome!

Adobe Flex Authors: Matthew Lobas, PR.com Newswire, Shelly Palmer, Kevin Benedict

Blog Feed Post

Monitoring HBase

HBase is a distributed, NoSQL, open-source database, initially conceived as an open-source alternative to Google’s proprietary BigTable. Originally, HBase was part of the Hadoop project, but was eventually spun off as a subproject. Given this legacy, it is not surprising that most often HBase is deployed on top of a Hadoop cluster (it used HDFS as its underlying storage), however a case study suggests that it can run on top of Amazon Elastic Block Store (EBS) as well. These days HBase is used by companies such as Adobe, Facebook, Twitter and Yahoo – and many others to process large amounts of data in real time, since it is ideally placed to store the input and/or the output of MapReduce jobs.

Monitoring HBase with Monitis and JMX

Like most products written in Java, both HBase and Hadoop contain built-in JMX instrumentation, which theoretically allows us to use any JMX client to view their performance metrics. Naturally, Monitis has just what the doctor ordered – a generic JMX agent that can be configured to monitor any JMX-enabled process through a point-and-click web interface.

Enabling JMX in HBase

By default the HBase JMX interface is disabled, but it can be enabled with relatively few configuration changes as explained in the official HBase documentation. (For security reasons and especially since we are not going to use JMX to modify the metric values or invoke administrative operations on the MBeans, I highly recommend that you do not add controlRole to jmxremote.passwd and jmxremote.access as the document suggests. Also, make sure these two files are owned by the login under which the HBase daemons will run, and that their permissions are 600, otherwise HBase will not start).

Installing the Monitis JMX Agent

The agent is implemented as a JEE web application and is packaged accordingly as a .war file. To download, log on to your Monitis account and select Monitors -> Manage Monitors -> JMX Monitors:

The JMX Monitors window will open. At the bottom of it, you will find a link to download the JMX Agent:

The .war file should be deployed in a standard JEE servlet container. While I recommend Tomcat due to its small footprint and ease of use, there are other options. If you already use an application server such as JBoss or WebSphere, you can deploy the JMX agent in it. While Tomcat offers many ways to deploy a .war file, the easiest one is to copy the .war file to Tomcat’s deploy folder. If Tomcat is already running, you don’t need to restart it – it will pick up and deploy the new .war file automatically.

Another consideration is which machine to deploy the monitor on. The HBase master would be the natural choice but your decision is going to be influenced by your specific network topology and corporate standards. In any case, the JMX agent should be able to accessthe HBase master on TCP port 10101 and optionally the Region Servers (data nodes) on port 10102. Aditionaly, Tomcat needs to be accessible on port 8080 (default).

Once the JMX agent .war file is deployed, go to http://<server_name>:8080/mon_jmx_agent. You should see the JMX agent’s login page which looks like this:

Enter your Monitis credentials – the same ones you use to login to your account on monitis.com – and click Login.

Creating an HBase Monitor in Monitis

Once it logs you in, the JMX Agent will prompt you to enter an Agent Name:

The Agent Name is used to uniquely identify the JMX Agent instance within Monitis. The metadata about the monitors is associated with your acount on monitis.com and the JMX agent will automatically download and run any existing monitors previously defined for this agent name. For  this reason, you want to choose a unique name for each JMX agent deployment. Once you enter a meaningful name and click Save, you should see the JMX Parameters page:

Make sure you enter the correct JMX port number and credentials you configured in HBase and click Submit to go to the next page:

Select the hadoop domain from the drop down – HBase’s MBeans live there for historical reasons. (You may also want to explore other domains – such as java.lang, which provides important information on JVM’s internals). Within the hadoop domain, select the HBase service -> RPC Statistics. The next screen shows an impressive number of metrics:

Under Monitor Name enter something meaningful. This is how your monitor will appear on monitis.com. Check interval is in minutes. Select the following attributes:

  • getNumOps
  • getAvgTime
  • getMinTime
  • getMaxTime
  • putNumOps
  • putAvgTime
  • putMinTime
  • putMaxTime

While most attribute names are self-explanatory, the MBean does not provide a meaningful description for the attributes, so feel free to examine the JMX section of the HBase book. Once you have selected all the metrics you are interested in, click on the Add Monitor button at the bottom of the page.

We are now ready to log on to Monitis and examine the data collected by our newly created monitor. If you are just logging on, Monitis will prompt you to add the new monitor, otherwise go to Monitors -> Manage Monitors -> JMX Monitors to open the familiar JMX Monitors screen:

Select the check box next to the new monitor and click Add to Window to open a new monitor window:

That’s it! As with any monitor, you can choose between multiple views and define notifications for your HBase performance metrics:

 

And finally, a few words about architecture. First, the JMX agent’s collector is implemented in the web application (war file). For this reason, you want to make sure that Tomcat (or whatever application server you deployed the agent’s war file on) keeps running for as long as you need to collect data. You may want to monitor the application server process itself and make sure it starts up automatically when the system is booted. Second, when you log on to the JMX Agent’s web interface, your credentials are submitted over an unencrypted HTTP connection (at least with the default Tomcat setup). This might be OK if you are viewing over the corporate LAN, otherwise you should look into enabling HTTPS on Tomcat. Alternatively, you could front Tomcat with Apache and let it do the heavy lifting.

In this installment we introduced the Monitis JMX Agent and hopefully convinced you how easy it is to monitor your HBase cluster. In a future post you will how you can use the agent to monitor JBoss. Happy monitoring!

 

Share Now:del.icio.usDiggFacebookLinkedInBlinkListDZoneGoogle BookmarksRedditStumbleUponTwitterRSS

Read the original blog entry...

More Stories By Hovhannes Avoyan

Hovhannes Avoyan is the CEO of PicsArt, Inc.,

@ThingsExpo Stories
trust and privacy in their ecosystem. Assurance and protection of device identity, secure data encryption and authentication are the key security challenges organizations are trying to address when integrating IoT devices. This holds true for IoT applications in a wide range of industries, for example, healthcare, consumer devices, and manufacturing. In his session at @ThingsExpo, Lancen LaChance, vice president of product management, IoT solutions at GlobalSign, will teach IoT developers how t...
A critical component of any IoT project is the back-end systems that capture data from remote IoT devices and structure it in a way to answer useful questions. Traditional data warehouse and analytical systems are mature technologies that can be used to handle large data sets, but they are not well suited to many IoT-scale products and the need for real-time insights. At Fuze, we have developed a backend platform as part of our mobility-oriented cloud service that uses Big Data-based approache...
We're entering the post-smartphone era, where wearable gadgets from watches and fitness bands to glasses and health aids will power the next technological revolution. With mass adoption of wearable devices comes a new data ecosystem that must be protected. Wearables open new pathways that facilitate the tracking, sharing and storing of consumers’ personal health, location and daily activity data. Consumers have some idea of the data these devices capture, but most don’t realize how revealing and...
When it comes to IoT in the enterprise, namely the commercial building and hospitality markets, a benefit not getting the attention it deserves is energy efficiency, and IoT's direct impact on a cleaner, greener environment when installed in smart buildings. Until now clean technology was offered piecemeal and led with point solutions that require significant systems integration to orchestrate and deploy. There didn't exist a 'top down' approach that can manage and monitor the way a Smart Buildi...
There is an ever-growing explosion of new devices that are connected to the Internet using “cloud” solutions. This rapid growth is creating a massive new demand for efficient access to data. And it’s not just about connecting to that data anymore. This new demand is bringing new issues and challenges and it is important for companies to scale for the coming growth. And with that scaling comes the need for greater security, gathering and data analysis, storage, connectivity and, of course, the...
The IETF draft standard for M2M certificates is a security solution specifically designed for the demanding needs of IoT/M2M applications. In his session at @ThingsExpo, Brian Romansky, VP of Strategic Technology at TrustPoint Innovation, will explain how M2M certificates can efficiently enable confidentiality, integrity, and authenticity on highly constrained devices.
So, you bought into the current machine learning craze and went on to collect millions/billions of records from this promising new data source. Now, what do you do with them? Too often, the abundance of data quickly turns into an abundance of problems. How do you extract that "magic essence" from your data without falling into the common pitfalls? In her session at @ThingsExpo, Natalia Ponomareva, Software Engineer at Google, will provide tips on how to be successful in large scale machine lear...
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus inter...
Digital payments using wearable devices such as smart watches, fitness trackers, and payment wristbands are an increasing area of focus for industry participants, and consumer acceptance from early trials and deployments has encouraged some of the biggest names in technology and banking to continue their push to drive growth in this nascent market. Wearable payment systems may utilize near field communication (NFC), radio frequency identification (RFID), or quick response (QR) codes and barcodes...
You think you know what’s in your data. But do you? Most organizations are now aware of the business intelligence represented by their data. Data science stands to take this to a level you never thought of – literally. The techniques of data science, when used with the capabilities of Big Data technologies, can make connections you had not yet imagined, helping you discover new insights and ask new questions of your data. In his session at @ThingsExpo, Sarbjit Sarkaria, data science team lead ...
SYS-CON Events announced today that Ericsson has been named “Gold Sponsor” of SYS-CON's @ThingsExpo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. Ericsson is a world leader in the rapidly changing environment of communications technology – providing equipment, software and services to enable transformation through mobility. Some 40 percent of global mobile traffic runs through networks we have supplied. More than 1 billion subscribers around the world re...
The demand for organizations to expand their infrastructure to multiple IT environments like the cloud, on-premise, mobile, bring your own device (BYOD) and the Internet of Things (IoT) continues to grow. As this hybrid infrastructure increases, the challenge to monitor the security of these systems increases in volume and complexity. In his session at 18th Cloud Expo, Stephen Coty, Chief Security Evangelist at Alert Logic, will show how properly configured and managed security architecture can...
The IoTs will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, will demonstrate how to move beyond today's coding paradigm and share the must-have mindsets for removing complexity from the development proc...
Artificial Intelligence has the potential to massively disrupt IoT. In his session at 18th Cloud Expo, AJ Abdallat, CEO of Beyond AI, will discuss what the five main drivers are in Artificial Intelligence that could shape the future of the Internet of Things. AJ Abdallat is CEO of Beyond AI. He has over 20 years of management experience in the fields of artificial intelligence, sensors, instruments, devices and software for telecommunications, life sciences, environmental monitoring, process...
In his session at @ThingsExpo, Chris Klein, CEO and Co-founder of Rachio, will discuss next generation communities that are using IoT to create more sustainable, intelligent communities. One example is Sterling Ranch, a 10,000 home development that – with the help of Siemens – will integrate IoT technology into the community to provide residents with energy and water savings as well as intelligent security. Everything from stop lights to sprinkler systems to building infrastructures will run ef...
We’ve worked with dozens of early adopters across numerous industries and will debunk common misperceptions, which starts with understanding that many of the connected products we’ll use over the next 5 years are already products, they’re just not yet connected. With an IoT product, time-in-market provides much more essential feedback than ever before. Innovation comes from what you do with the data that the connected product provides in order to enhance the customer experience and optimize busi...
Manufacturers are embracing the Industrial Internet the same way consumers are leveraging Fitbits – to improve overall health and wellness. Both can provide consistent measurement, visibility, and suggest performance improvements customized to help reach goals. Fitbit users can view real-time data and make adjustments to increase their activity. In his session at @ThingsExpo, Mark Bernardo Professional Services Leader, Americas, at GE Digital, will discuss how leveraging the Industrial Interne...
The increasing popularity of the Internet of Things necessitates that our physical and cognitive relationship with wearable technology will change rapidly in the near future. This advent means logging has become a thing of the past. Before, it was on us to track our own data, but now that data is automatically available. What does this mean for mHealth and the "connected" body? In her session at @ThingsExpo, Lisa Calkins, CEO and co-founder of Amadeus Consulting, will discuss the impact of wea...
Increasing IoT connectivity is forcing enterprises to find elegant solutions to organize and visualize all incoming data from these connected devices with re-configurable dashboard widgets to effectively allow rapid decision-making for everything from immediate actions in tactical situations to strategic analysis and reporting. In his session at 18th Cloud Expo, Shikhir Singh, Senior Developer Relations Manager at Sencha, will discuss how to create HTML5 dashboards that interact with IoT devic...
Whether your IoT service is connecting cars, homes, appliances, wearable, cameras or other devices, one question hangs in the balance – how do you actually make money from this service? The ability to turn your IoT service into profit requires the ability to create a monetization strategy that is flexible, scalable and working for you in real-time. It must be a transparent, smoothly implemented strategy that all stakeholders – from customers to the board – will be able to understand and comprehe...