Uploaded image for project: 'Sample Applications'
  1. Sample Applications
  2. APP-107

Basic level Apps: Data collection

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 2.1.0
    • Component/s: Sample app: common
    • Labels:
    • Epic Name:
      Basic level Apps: Data collection

      Description

      The below description is outdated.

      For updated scenario flow see APP-315;
      For updated bundle description see APP-317.


      As a platform user I would like to have a very basic sample application that would demonstrate Kaa Logging extension functionality with detailed description. The application must be available in all supported languages and illustrate the use of Data collection management APIs on the client side, as well as the use of Administrative UI and the REST interfaces for managing Log schemas and Log appenders.

      h1. Scenario description

      This application bundle will demonstrate capability to collect measurements from temperature sensor from the client into MongoDB. The Log schema will contain temperature and time-stamp properties:

      {
        "type":"record",
        "name":"LogData",
        "namespace":"org.kaaproject.kaa.schema.sample.logging",
        "fields":[
          {
            "name":"temperature",
            "type":"int"
          },
          {
            "name" : "timeStamp",
            "type" : "long"
          }
        ]
      }
      


      The client application will be generating log records, send them to server and printing log delivery status for every particular log record.
      [email protected]:~/$ java -jar DataCollectionDemo.jar 
      2016-05-31 12:45:13,982 [main] INFO  o.k.k.d.d.DataCollectionDemo - Data collection demo started
      2016-05-31 12:45:14,989 [pool-2-thread-1] INFO  o.k.k.d.d.DataCollectionDemo - Kaa client started
      2016-05-31 12:45:14,995 [main] INFO  o.k.k.d.d.DataCollectionDemo - Log record {"temperature": "10", "timeStamp": 1464687914993} sent
      2016-05-31 12:45:14,996 [main] INFO  o.k.k.d.d.DataCollectionDemo - Log record {"temperature": "20", "timeStamp": 1464687914993} sent
      2016-05-31 12:45:14,996 [main] INFO  o.k.k.d.d.DataCollectionDemo - Log record {"temperature": "15", "timeStamp": 1464687914993} sent
      2016-05-31 12:45:14,996 [main] INFO  o.k.k.d.d.DataCollectionDemo - Log record {"temperature": "-10", "timeStamp": 1464687914993} sent
      2016-05-31 12:45:14,997 [main] INFO  o.k.k.d.d.DataCollectionDemo - Log record {"temperature": "15", "timeStamp": 1464687914993} sent
      2016-05-31 12:45:17,143 [main] INFO  o.k.k.d.d.DataCollectionDemo - Received log record delivery info. Bucket Id [0]. Record delivery time [2150 ms].
      2016-05-31 12:45:17,144 [main] INFO  o.k.k.d.d.DataCollectionDemo - Received log record delivery info. Bucket Id [0]. Record delivery time [2150 ms].
      2016-05-31 12:45:17,144 [main] INFO  o.k.k.d.d.DataCollectionDemo - Received log record delivery info. Bucket Id [0]. Record delivery time [2150 ms].
      2016-05-31 12:45:17,144 [main] INFO  o.k.k.d.d.DataCollectionDemo - Received log record delivery info. Bucket Id [0]. Record delivery time [2150 ms].
      2016-05-31 12:45:17,144 [main] INFO  o.k.k.d.d.DataCollectionDemo - Received log record delivery info. Bucket Id [0]. Record delivery time [2150 ms].
      2016-05-31 12:45:17,146 [pool-2-thread-1] INFO  o.k.k.d.d.DataCollectionDemo - Kaa client stopped
      2016-05-31 12:45:17,154 [main] INFO  o.k.k.d.d.DataCollectionDemo - Data collection demo stopped
      


      The application will use MongoDB Log appender and trustful credentials service for simplicity. User will be able to observe delivered data on live dashboard in a separate specially developed web application on the Sandbox VM.



      This demo also contains configuration schema to control the log count threshold in Log upload strategy (how many log records must be collected before send them to Kaa node):
      {
        "type" : "record",
        "name" : "LogUploadThreshold",
        "namespace" : "org.kaaproject.example.datacollection",
        "fields" : [ {
          "name" : "Threshold",
          "type" : "int",
          "by_default" : 1
        } ],
        "displayName" : "Logging configuration schema"
      }
      


      The default log count threshold is 1. It means that every added log will be sent immediately. Use Administrative UI for managing Log upload threshold: just change the threshold value in corresponding configuration inside All group.

      h2. Scenario flow

      In this application we have a few components:
      # Client application
      # Kaa node
      # Separate web app for log delivery visualization

      and next demonstration scenario:
      # User starts the Client application.
      # Client application: sends log records with randomly generated values of temperature in an infinite loop, receives log delivery confirmation and prints it to the console.
      # Web app connects to DB to fetch data for time line diagram rendering
      # User goes to Web app to ensure that log records were saved

      List of links on documentation that must be present in application description:
      # http://docs.kaaproject.org/display/KAA/Endpoint+registration
      # http://avro.apache.org/docs/current/spec.html#schemas
      # http://docs.kaaproject.org/display/KAA/Logging
      # http://docs.kaaproject.org/display/KAA/Collecting+data+from+endpoints
      # http://docs.kaaproject.org/display/KAA/Supported+log+appenders
      # http://docs.kaaproject.org/display/KAA/Creating+custom+log+appender
      # http://docs.kaaproject.org/display/KAA/Administration+UI+guide
      # http://docs.kaaproject.org/display/KAA/Admin+REST+API#AdminRESTAPI-Logging

      h1. Proposed description for application inside sandbox (incomplete)

      h2. Introduction

      This bundle contains applications that demonstrates capabilities of the Kaa Logging subsystem.
      (paste here more "water" about how important this feature is for IoT network)
      The Kaa Logging subsystem allows to collect log records of pre-configured structure from the endpoints, periodically transferring these logs from endpoints to Operation servers, and, finally, either persisting them on the server for further processing or submitting them to immediate stream analysis.



      The Kaa logs structure is determined by the configurable log schema. In this demonstration application we are gathering values of measurements from some temperature sensor and saving them into MongoDB. Data format for log records we have defined by the next schema:
      You can find this schema in Administrative console - log in as a tenant developer (devuser / devuser123), and navigate to the "Data collection demo" application.
      Than you need to select "Schemas" and than "Log". Here you can find list of available schemas for your application.
      As you may guess "Log schema" - is the schema that we used in this demo. If you click on it to overview the structure of schema.
      Also we are using MongoDB Log appender, Log appenders are responsible for writing logs received by the Operations server into specific storage systems. If you choose "Log appenders" tab in Administrative console you will finds the list of log appenders for your application.
      And if you click on "Data collection log appender" you will find configuration page for given appender, where you can for example change database host.
      Configuration for this Log appender you can find in our GitHub repository.
      Also in endpoint application we are setting Log upload strategy. We selected RecordCountLogUploadStrategy with threshold 1 - following this strategy endpoint will be sending every log record separately. For more information refer to documentation.
              DOTO: reference on github code
      


      So lets consider whats going on in this application.

      h2. Playing around

      Playing around section: must contain some examples for modification of the given application. For example, provide description how to modify Log record schema or how to add REST Log appender and send data on web service, provide links for other Log appenders and short overview of their abilities.

      Also provide some information about the ways to fetch data from databases.

      # Use configuration feature for managing Log upload threshold: you can control upload strategy threshold using Configuration feature. Just change threshold value for the appropriate configuration in group "All" and activate it. You can also load new configuration from file. The file format must be:
      {
        "Threshold" : 4
      }
      

      Demo application is already subscribed on configuration updates and you can see changed logs upload threshold immidietly in running application.
      # Add extra Log appender, for example REST log appender, first without "confirm delivery" option enabled (we have no any web app that will consume this logs) to demonstrate that application will ignore fails of REST log appender to save logs, and then with this option enabled - demonstrate that endpoint wouldn't delete any logs until REST appender succeed to persist logs.

      h2. What's next?

      For more background on the Kaa Logging subsystem, you can check out "Cell monitor demo" and also refer to the corresponding programming guide and design reference sections.

        Attachments

        1. log_delivery_chain.png
          log_delivery_chain.png
          20 kB
        2. log_delivery_chain.xml
          2 kB
        3. log_delivery_websequence.txt
          2 kB
        4. Log delivery sequence .png
          Log delivery sequence .png
          77 kB
        5. time_line_example.png
          time_line_example.png
          49 kB

          Issue Links

            Activity

              People

              Assignee:
              akokhanovskyi Andrew Kokhanovskyi [Administrator]
              Reporter:
              krupet Peter Krutsiuk
              Responsible:
              Andrew Kokhanovskyi [Administrator] Andrew Kokhanovskyi [Administrator]
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration