Differences

This shows you the differences between two versions of the page.

--- external_api_data [2023/01/26 09:32] – created angelos
+++ external_api_data [2023/11/21 16:12] (current) – chris
@@ Line 3: / Line 3: @@
 Oftentimes you have external data coming from a REST HTTP API and want access to it through the Pathomation software platform for digital pathology and virtual microscopy. Importing the data as Pathomation form data may not always make sense for a variety of reasons.
-PMA.core allows you to link external rest APIs and use them as regular form data.
+PMA.core allows you to link external rest APIs and use them as regular form data. This works by invoking a user-defined command line application which must exist in the system that hosts PMA.core. Such applications can be:
+  * curl
+  * python
+  * custom cli utilities
-==== Configuring data connections ====
+PMA.core executes the application for a given image path and can also supply extra command line arguments. Once the application has finished, PMA.core reads the standard output stream and expects a JSON response, in which it attempts to read values for particular fields.
-The first step in configuring your external data link to PMA.core is to create the connection to the data source. PMA.core supports connection to the following databases
+==== Configuring HTTP API data connections ====
-  * SQL Server
-  * Oracle
+To configure your external HTTP API data link to PMA.core, click on the //Add// button in the **External API data** section of the External Data page, of the administrator front-end UI.
-{{ :external_connection_add.png?600 |}}
-To add an external connection you simply click on the //Add// button in ExternalData page of the administrator front-end UI and then select the type of connection you want. On the next page you will be prompted to add
+{{:external-api-data-add.png?600|}}
+On the next page you will be prompted to add
   * Name: A name for the new connection to distinguish this connection from others
-  * Connection String: The connection string to the database system including the credentials PMA.core will use to connect
+  * Command: The name or path of the application to execute
+  * Arguments: Command line arguments to pass to the application
+  * Fields: Field names to expect in the JSON output, once the command is executed
-{{ ::externa_connection_add_page.jpg?600 |}}
+{{:add-external-api-data-connection.png?direct&600|}}
-==== Configure queries ====
+=== Arguments ===
-After creating the external database connection we need to create a link between the data and the slides in PMA.core. To accomplish this we need to define an SQL query that will instruct PMA.core how each slide is linked to one or more rows of data in the external connection. To start that we need to click ton Add button in the external data source page
+In the arguments field you can specify any number of command line arguments that are required for the correct execution of the command. For example, if you want to run a python script, the value for the command should be "python" and the arguments should be the name of the script, e.g. "my-script.py".
-=== SQL query ===
+Additionally, PMA.core can identify the following keywords in the arguments string and replace them with their actual values whenever the REST API data source is consulted:
-{{ ::add_external_data.jpg?600 |}}
+  - [path]: Gets replaced with the path of the image for which the external API is consulted.
-Even though the Add Query page seems complicated it is nothing more than an intuitive UI to create an SQL query so that PMA.core can link the the external data to the slides provided. The page is divided in two parts, on the right hand side there is a treeview with PMA.core slides that is only used for testing the query and on the left hand side UI to create the query. The parts that are needed to successfully create the query are:
+  - [filename]: Gets replaced with the filename of the image for which the external API is consulted.
-  * Connection: One of the external connections created earlier (see previous paragraph)
+  - [filename_no_ext]: Gets replaced with the filename, without the extension, of the image for which the external API is consulted.
-  * Table name: This field is **automatically** populated after you select the //Connection//. This is the table with the data you want to link to PMA.core
+  - [barcode]: Gets replaced with the barcode text, if any, of the image for which the external API is consulted.
-  * Display name: A name for this query, just to distinguish between this and other queries
+  - [oidc]: Gets replaced with the calling user's OpenID token, if applicable.
-  * Fields to include: This is a list of all fields in the table. It is **automatically** populated after selecting the //Table Name//.  You can then edit this to remove any fields you do not want. To reset this you simply need to reselect the //Table Name//.
+  - [session_id]: Gets replaced with the calling user's session id.
-  * Match mode: PMA.core offers two ways to link the external data to PMA.core slides. This modes are **match based on file name** or **match based on the barcode text**. This is the value that will be passed on the query to select data based upon, and in the Query Preview field is identified as **#SLIDE_IDENTIFIER#**
-  * Regular expression: This is an //optional// regular expression that will be applied on **SLIDE_IDENTIFIER** before passing it to the query. This way you can transform the filename or barcode to match what is expected by the database (e.x. removing the path, the filename extension etc)
-  * Filter Expression: This is the //WHERE// expression of the query, that have to match the **SLIDE_IDENTIFIER**. This can simply be a column of the table or some other valid SQL WHERE expression
-  * Query preview: This is an important preview of the final query. After each change to the previous fields your changes will be reflected here for final inspection.
-As stated an above the **#SLIDE_IDENTIFIER#** part of the query is dependent on the mode selected and the regular expression if provided. So this can be the file name or the barcode text of a slide, after it is transformed by the regular expression.
+For example, if you have configured PMA.core to allow logging in via an OAuth 2.0 SSO provider, you can pass the user's access token to the external API, supposing it uses the same authentication provider. Here's a curl example:
-Before your query is ready you need to test that it works as expected
+  * Command: curl
+  * Arguments: -G -X GET --header "Accept: application/json" "https://my-api-url" --data-urlencode "token=[oidc]" --data-urlencode "pathOrUid=[path]"
+=== Fields ===
+PMA.core can read top level field values from a JSON response. You can specify a list of fields it should attempt to read, each separated in a new line.
 === Testing ===
-To test your query for errors and validating its results you can use the Test query field. You can either write a slide identifier manually in the text box or use the //Set from treeview// button after selecting a slide from the treeview on the right hand side of the screen.
+You can test the result of the command from this page. Simply fill out an image path for which the command should run and PMA.core will display the raw results of the command in a pop up:
-{{ :test_query.jpg?400 |}}
+{{:external-api-data-raw-command.png?direct&400|}}
-Clicking on the Execute button will execute the query on the server. Any errors in syntax will be reported immediately bellow the execute button. If no errors occur a popup will appear that will show the results of the query.
-{{ ::test_query_popup.jpg?400 |}}
 ==== Consumption and applications ====
-After creating the queries you can now use the [[https://docs.pathomation.com/pma.core.api/#/Api/PMA.Services.Api.HistoScope.GetFormData|FormData]] API to request data by using the Query Id as the Form Id, the same way you request native PMA.core Form Data.
+After creating the connection you can now use the [[https://docs.pathomation.com/pma.core.api/#/Api/PMA.Services.Api.HistoScope.GetFormData|FormData]] API to request data by using the Query ID as the Form ID, the same way you request native PMA.core Form Data.
-This is the same API that is used to provide external data in downstream application like [[https://www.pathomation.com/pma.studio|PMA.studio]], [[https://www.pathomation.com/pma.slidebox|PMA.slidebox]], [[https://www.pathomation.com/pma.control|PMA.control]] etc.
+This is the same API that is used to provide external data in downstream applications like [[https://www.pathomation.com/pma.studio|PMA.studio]], [[https://www.pathomation.com/pma.slidebox|PMA.slidebox]], [[https://www.pathomation.com/pma.control|PMA.control]] etc.
 {{ ::pmastudio_external_data.jpg?600 |}}
-==== More background ====
-See [[https://realdata.pathomation.com/how-to-handle-slide-meta-data/|our blog article]].