Azure logs analysis |
[This is preliminary documentation and is subject to change.]
This topic contains the following sections:
The purpose of this tutorial is to help you understand how to build an “Azure IIS Log Analytics” application with Eety.
Create an application solution:
|
Azure provides built-in diagnostics to assist with debugging an App Service web app. You can enable Web Server Logging - Information about HTTP transactions.
Diagnostic information stored to the web app file system can be accessed directly using FTP. The /LogFiles/http/RawLogs folder contains one or more text files formatted using the W3C extended log file format.
Eety provides an FTP connector to download files and a Delimited Text file connector to load them.
To create an FTP connection:
|
To create an Delimited Text connection:
Right click on Connections node, in Solution Explorer pane.
Expand the Add connection menu.
Select Delimited Text File.
Fill the dialog box with the relevant values:
File Tab
Parameters | Values |
---|---|
Name | Logs |
File | Logs\*.log |
Schema path | |
Table Name | Hit |
Culture | English |
Schema on first row | False |
Text Qualifier | True |
Column delimiter | Whitespace |
Row delimiter | CRLF |
Comments | # |
Columns Tab
Column | Type | Format | Nullable |
---|---|---|---|
date | DateTime | yyyy-MM-dd | |
time | TimeSpan | ||
s-sitename | String | ||
cs-method | String | ||
cs-method | String | ||
cs-uri-query | String | ||
s-port | Int32 | ||
cs-username | String | ||
c-ip | String | ||
cs(User-Agent) | String | ||
cs(Cookie) | String | ||
cs(Referer) | String | ||
cs-host | String | ||
time | TimeSpan | ||
sc-status | Int32 | ||
sc-substatus | Int32 | ||
sc-win32-status | Int32 | ||
sc-bytes | Int32 | ||
cs-bytes | Int32 | ||
time-taken | Int32 |
One single model shares its elements with all the components in the application. This is a logical business model with tables, columns and relationships mapped to the data sources corresponding elements. At runtime, data extracted from the data sources is stored into these elements, starting an integration or analysis pipeline.
Note |
---|
In this example, all logs files will be extracted to a memory table named Hit |
Right click on the Mappings node, in the Solution Explorer pane.
Add a new Extraction type mapping.
Optionally, double click the New mapping node in the Solution Explorer pane.
Optionally, click to open the help popup.
Click on the create and map a new transfer model to the selected data source label.
Note |
---|
Right click on the New mapping node in Solution Explorer (or select it and press F2) and rename it Log Files. |
To add an analysis workflow:
|
A trigger is a parameterized conditional procedure automatically executed on data flow events.
Log file must be downloaded from the FTP server before to start the extraction phase
We need to concatenate the date and time fields.
Expand the Analysis node in the Solution Explorer pane and double click on the Extraction node.
Select the Logs source in the left pane tree of the Extraction Stage.
Double click on button in the bottom toolbar of the center pane.
Click the button on the right of Download files in the listbox.
Input parameters as shown below:
Optionally, expand the Analysis in the Solution Explorer pane and double Click on the Extraction node.
Expand the Logs source in the left pane tree of the Extraction Stage and select the Hit table.
Double click the button in the bottom toolbar of the center pane.
Click the button on the right of Computed values in the listbox.
Add the date column and click on function button:
In the Formula Editor, optionally click on the Variable button at the bottom.
Double click on the date label in the Variables listbox. This adds a V1 expression in the formula line.
Click on the + button.
Double click on the time label in the Variables listbox. This adds a V2 expression in the formula line.
Click on the Ok button.
We want to aggregate the Hits into a Pages table.
Technical Definitions:
A page view is a request from a web surfer to load a single web page.
A hit is a request to a web server for a file. There may be many hits per page view.
Expand the Analysis node in Solution Explorer pane.
Double Click Transformation.
Click Hits in the model tree on the left pane.
Click Add new button on the right of the toolbar.
Select Aggregation in the Wizards listbox.
Input Page in the table name textbox.
Expand the From group and select s-sitename in the drop down list.
Expand the Operation group and select GroupBy in the drop down list.
Click the button.
Similarly, add the cs-host , s-port, cs-method and cs-uri-stem group by columns.
Select * in the From drop down list.
Select Count in the Operation drop down list.
Click the button.
Click the button. Rename the columns and click the Ok button.
Click the Ok button.
We want to release all the analysis reports as html views and browse them into a dedicated web site.
Expand the Analysis node in Solution Explorer pane.
Double Click Report.
Click Logs in the model tree on the left pane.
Click Add new button on the right of the toolbar.
Select Web Application on the left of the Reporting dialog box.
Click the site templates button on the bottom of the box.
Select the jquery+bootstrap+highcharts.etpl template file in the dialog box.
Click Open.
Fill the dialog box with the relevant values:
Click OK.
Click Invoices in the model tree on the left pane of the Reporting stage page.
Double click the button in the bottom toolbar of the center pane.
Click the button on the right of Start Application in the listbox.
Input parameters as shown below:
Expand the Analysis node in Solution Explorer pane.
Double Click Report.
Expand Logs in the model tree on the left pane.
Select Page.
Click Add new button on the right of the toolbar.
Select Html View on the left of the Reporting dialog box.
Click the site templates button on the bottom of the box.
Select the jquery+bootstrap+highcharts.etpl template file in the dialog box.
Click Open.
Fill the dialog box with the relevant values:
Click OK.
Click in Solution Explorer toolbar.
Click in Test New Application.
A default browser instance should be started.