Click or drag to resize

Azure logs analysis

[This is preliminary documentation and is subject to change.]

ABOUT THIS GUIDE

The purpose of this tutorial is to help you understand how to build an “Azure IIS Log Analytics” application with Eety.

GETTING STARTED

Create an application solution:

  1. Launch Eety Desktop.

  2. Click NEW in the file toolbar.

  3. Optionally, change the destination folder in the file dialog box.

  4. Fill in a name for the solution file (without any .eety extension) in the file name text box.

  5. Click Open.

    The solution workspace opens its Solution Explorer on the right side and a start page in a New Solution window.

    This start page simply depicts steps to walk through building an application:

    • Add source and target connections

    • Map sources and targets with the data transfer model

    • Add an analysis workflow

openfiledialog
STEP 1: ADD SOURCE AND TARGET CONNECTIONS

ADD DELIMITED TEXT FILES CONNECTION

  • Azure provides built-in diagnostics to assist with debugging an App Service web app. You can enable Web Server Logging - Information about HTTP transactions.

  • Diagnostic information stored to the web app file system can be accessed directly using FTP. The /LogFiles/http/RawLogs folder contains one or more text files formatted using the W3C extended log file format.

  • Eety provides an FTP connector to download files and a Delimited Text file connector to load them.

    To create an FTP connection:

    1. Right click on the Storages node the in Solution Explorer pane.

    2. Expand the Add Storage menu.

    3. Select the FTP connector.

    4. Fill the dialog box with the relevant values.

    Example 2 FTP 1
    Example 2 FTP 2
  • To create an Delimited Text connection:

    1. Right click on Connections node, in Solution Explorer pane.

    2. Expand the Add connection menu.

    3. Select Delimited Text File.

    4. Fill the dialog box with the relevant values:

      File Tab

      Parameters

      Values

      Name

      Logs

      File

      Logs\*.log

      Schema path

      Table Name

      Hit

      Culture

      English

      Schema on first row

      False

      Text Qualifier

      True

      Column delimiter

      Whitespace

      Row delimiter

      CRLF

      Comments

      #

      Columns Tab

      Column

      Type

      Format

      Nullable

      date

      DateTime

      yyyy-MM-dd

      time

      TimeSpan

      s-sitename

      String

      cs-method

      String

      cs-method

      String

      cs-uri-query

      String

      s-port

      Int32

      cs-username

      String

      c-ip

      String

      cs(User-Agent)

      String

      cs(Cookie)

      String

      cs(Referer)

      String

      cs-host

      String

      time

      TimeSpan

      sc-status

      Int32

      sc-substatus

      Int32

      sc-win32-status

      Int32

      sc-bytes

      Int32

      cs-bytes

      Int32

      time-taken

      Int32

    Example 2 CSV 1
    Example 2 CSV 2
STEP 2: MAP SOURCES AND TARGETS WITH THE LOGICAL MODEL

Basic concepts

One single model shares its elements with all the components in the application. This is a logical business model with tables, columns and relationships mapped to the data sources corresponding elements. At runtime, data extracted from the data sources is stored into these elements, starting an integration or analysis pipeline.

Note Note

In this example, all logs files will be extracted to a memory table named Hit

To create the extraction mapping

  1. Right click on the Mappings node, in the Solution Explorer pane.

  2. Add a new Extraction type mapping.

To create and map a new model

  1. Optionally, double click the New mapping node in the Solution Explorer pane.

  2. Optionally, click show-help 24x 24to open the help popup.

  3. Click on the create and map a new transfer model to the selected data source label.

Example 2 Mapping
Note Note

Right click on the New mapping node in Solution Explorer (or select it and press F2) and rename it Log Files.

STEP 3: ADD AN ANALYSIS WORKFLOW

To add an analysis workflow:

  1. Right click on Workflows, in Solution Explorer.

  2. Add an analysis worfklow.

  3. Right click the new workflow node and rename it Analysis.

Example 2 Workflow
STEP 4: SET UP THE EXTRACTION PHASE

Basic concepts

A trigger is a parameterized conditional procedure automatically executed on data flow events.

  • Log file must be downloaded from the FTP server before to start the extraction phase

  • We need to concatenate the date and time fields.

To add a download trigger:

  1. Expand the Analysis node in the Solution Explorer pane and double click on the Extraction node.

  2. Select the Logs source in the left pane tree of the Extraction Stage.

  3. Double click on trigger-download button in the bottom toolbar of the center pane.

    Example 2 Extraction FTP 1
  4. Click the edit-delegate 16x 16 button on the right of Download files in the listbox.

  5. Input parameters as shown below:

    Example 2 Extraction FTP 2

To add a Computed Values trigger:

  1. Optionally, expand the Analysis in the Solution Explorer pane and double Click on the Extraction node.

  2. Expand the Logs source in the left pane tree of the Extraction Stage and select the Hit table.

  3. Double click the trigger-formula button in the bottom toolbar of the center pane.

    Example 2 Extraction Formula 1
  4. Click the edit-delegate 16x 16 button on the right of Computed values in the listbox.

  5. Add the date column and click on function button:

    Example 2 Extraction Formula 2
  6. In the Formula Editor, optionally click on the Variable button at the bottom.

  7. Double click on the date label in the Variables listbox. This adds a V1 expression in the formula line.

  8. Click on the + button.

  9. Double click on the time label in the Variables listbox. This adds a V2 expression in the formula line.

  10. Click on the Ok button.

    Example 2 Extraction Formula 3
STEP 5: SET UP THE TRANSFORMATION PHASE

Basic concepts

We want to aggregate the Hits into a Pages table.

Technical Definitions:

  • A page view is a request from a web surfer to load a single web page.

  • A hit is a request to a web server for a file. There may be many hits per page view.

To add an aggregation

  1. Expand the Analysis node in Solution Explorer pane.

  2. Double Click Transformation.

  3. Click Hits in the model tree on the left pane.

  4. Click Add new button on the right of the toolbar.

  5. Select Aggregation in the Wizards listbox.

  6. Input Page in the table name textbox.

  7. Expand the From group and select s-sitename in the drop down list.

  8. Expand the Operation group and select GroupBy in the drop down list.

  9. Click the add-element 16x 16 button.

    Example 2 Transform Aggregation 1
  10. Similarly, add the cs-host , s-port, cs-method and cs-uri-stem group by columns.

    Example 2 Transform Aggregation 2
  11. Select * in the From drop down list.

  12. Select Count in the Operation drop down list.

  13. Click the add-element 16x 16 button.

    Example 2 Transform Aggregation 3
  14. Click the trigger-default button. Rename the columns and click the Ok button.

    Example 2 Transform Aggregation 4
  15. Click the Ok button.

STEP 6: CREATE A REPORTING WEB APPLICATION

Basic concepts

We want to release all the analysis reports as html views and browse them into a dedicated web site.

CREATE A WEB SITE

To create a web application

  1. Expand the Analysis node in Solution Explorer pane.

  2. Double Click Report.

  3. Click Logs in the model tree on the left pane.

  4. Click Add new button on the right of the toolbar.

    Example 2 Reporting Site 1
  5. Select Web Application on the left of the Reporting dialog box.

  6. Click the site templates button on the bottom of the box.

  7. Select the jquery+bootstrap+highcharts.etpl template file in the dialog box.

    Example 2 Reporting Site 2
  8. Click Open.

  9. Fill the dialog box with the relevant values:

    Example 2 Reporting Site 3
  10. Click OK.

To add an auto start

  1. Click Invoices in the model tree on the left pane of the Reporting stage page.

  2. Double click the reporting-webstart button in the bottom toolbar of the center pane.

    Example 2 Reporting Start 1
  3. Click the edit-delegate 16x 16 button on the right of Start Application in the listbox.

  4. Input parameters as shown below:

    Example 2 Reporting Start 2

CREATE THE HTML VIEWS

To create an html view

  1. Expand the Analysis node in Solution Explorer pane.

  2. Double Click Report.

  3. Expand Logs in the model tree on the left pane.

  4. Select Page.

  5. Click Add new button on the right of the toolbar.

    Example 2 Reporting Site 1
  6. Select Html View on the left of the Reporting dialog box.

  7. Click the site templates button on the bottom of the box.

  8. Select the jquery+bootstrap+highcharts.etpl template file in the dialog box.

    Example 2 Reporting Site 2
  9. Click Open.

  10. Fill the dialog box with the relevant values:

    Example 2 Reporting Site 3
  11. Click OK.

STEP 7: TEST THE APPLICATION

To test the application

  1. Click debugger 16x 16 in Solution Explorer toolbar.

  2. Click workflow-start 16x 16 in Test New Application.

  3. A default browser instance should be started.