Transform derived integration and derived validation

README.md

@@ -414,8 +414,8 @@ Artesian support Query over GME Public Offers which comes in a custom and dedica
 Note (performance):
 GME Public Offer data is partitioned by date, offerType, status, and market. Requesting very narrow subsets (for example a single status or offerType in several separate requests) does not improve performance and can cause the same data to be fetched multiple times.
 For this reason, the examples below:
-	•	Use a large page size so that all data for a given (date, market, filters) is typically returned in a single page.
-	•	Loop over markets explicitly to cover all required markets without redundant fetches.
+• Use a large page size so that all data for a given (date, market, filters) is typically returned in a single page.
+• Loop over markets explicitly to cover all required markets without redundant fetches.
 
 ### Extract GME Public Offer
 
@@ -708,6 +708,76 @@ Extraction options for GME Public Offer queries.
  .withAggregationRule(AggregationRule.SumAndDivide)
 ```
 
+### DerivedTransform: Query Validation
+
+Use the `DerivedTransformQueryValidation` to validate and execute a derived transform query against a sample TimeSerieData.
+
+```Python
+from Artesian.MarketData import MarketDataService
+from Artesian.MarketData._Dto.DerivedTransformQueryValidation import DerivedTransformQueryValidation
+from Artesian.MarketData._Dto.TimeSerieData import TimeSerieData
+from Artesian import MarketDataType
+from datetime import datetime
+
+mds = MarketDataService(cfg)
+
+request = DerivedTransformQueryValidation(
+    data=TimeSerieData(
+        rows=[
+            (datetime(2018, 10, 1, 0, 0), 100),
+            (datetime(2018, 10, 1, 1, 0), 100)
+        ],
+        type=MarketDataType.ActualTimeSerie,
+    ),
+    transform="SELECT Time, (Value + 1) as Value FROM $table"
+)
+
+derivedTransformResponse = mds.derivedTransformQueryValidation(request)
+```
+
+#### Available Columns
+
+| Column | Type     |
+| ------ | -------- |
+| Time   | datetime |
+| Value  | double   |
+
+> `$table` is a virtual table exposed by the query engine.
+
+**Returned Object: DerivedTransformQueryValidationResponse**
+
+| Property | Description                                    |
+| -------- | ---------------------------------------------- |
+| Data     | Transformed time series generated by the query |
+| Valid    | Indicates whether the query is valid           |
+| Error    | Validation error details when `Valid = false`  |
+
+#### Query examples
+
+```sql
+SELECT Time + INTERVAL 1 DAY AS Time, Value FROM $table
+
+SELECT Time, CASE WHEN EXTRACT(HOUR FROM (Time AT TIME ZONE 'UTC') AT TIME ZONE 'Europe/Rome') < 10 THEN Value + 1 ELSE Value END AS Value FROM $table WHERE Time IS NOT NULL
+
+SELECT Time, Value FROM $table WHERE Version IS NOT NULL AND ((EXTRACT(hour FROM Version) < 10 AND Time >= date_trunc('day', Version + interval '1 day')) OR (EXTRACT(hour FROM Version) >= 10 AND Time >= date_trunc('day', Version + interval '2 day')))
+```
+
+#### SQL Dialect & Execution Model
+
+The query engine uses a DuckDB-compatible SQL dialect (PostgreSQL-like).
+
+Queries are executed against a virtual table named `$table`, which represents the provided sample TimeSerieData.
+
+The engine supports common SQL features including:
+
+- SELECT expressions
+- WHERE filtering
+- CASE WHEN expressions
+- Date/time arithmetic and functions
+- Timezone conversion functions
+
+Full DuckDB SQL reference: https://duckdb.org/docs/sql/introduction.html
+
 ## Write Data in Artesian
 
 Using the MarketDataService is possible to register MarketData and write curves into it using the UpsertData method.
@@ -761,22 +831,23 @@ mkservice.upsertData(data)
 ```
 
 Upsert has optional switches that can be applied
+
 ```
 mkservice.upsertData(data, deferCommandExecution, deferDataGeneration, keepNulls, upsertMode)
 ```
+
 The switch details are,
 
 deferCommandExecution (true/false) choose between syncronoys and asyncronous command execution, default is false.
 deferDataGeneration (true/false) choose between syncronoys and asyncronous precomputed data generation, default is true.
 keepNulls (true/false) if true then nulls are written in the curve replacing any data present for the instant, default is false.
 upsertMode (Merge/Replace) for ActualTimeSeries the two modes are equivalent. Leaving Null/None/Empty is equivalent to Merge.
 
-
-DerivedCfg can be of algorithm type: Coalesce, Sum, Muv.
+DerivedCfg can be of algorithm type: Coalesce, Sum, Muv, Transform.
 
 Updating the DerivedCfg can be performed with `updateDerivedConfiguration` on MarketDataService. A validation will be done on the existing DerivedCfg of the MarketData, that should be not null and with same type as the one used for the update.
 
-```csharp
+```Python
 derivedCfgUpdate = DerivedCfg(
     version=1,
     derivedAlgorithm=DerivedAlgorithm.Coalesce,
@@ -789,6 +860,31 @@ marketDataUpdated = mkdservice.updateDerivedConfiguration(
                         False)
 ```
 
+For DerivedCfg Transform, `orderedReferencedMarketDataIds` contains a single source series (the series where the transform is applied), and `transform` contains the query to apply.
+
+```Python
+derivedCfgUpdate = DerivedCfg(
+    version=1,
+    derivedAlgorithm=DerivedAlgorithm.Transform,
+    orderedReferencedMarketDataIds=[10001],
+    transform="SELECT Time, (Value + 1) as Value FROM $table"
+)
+
+marketDataUpdated = mkdservice.updateDerivedConfiguration(
+                        registeredDerived.marketDataId,
+                        derivedCfgUpdate,
+                        False)
+```
+
+#### Available Columns
+
+| Column | Type     |
+| ------ | -------- |
+| Time   | datetime |
+| Value  | double   |
+
+> `$table` is a virtual table exposed by the query engine.
+
 In case we want to write an hourly (or lower) time series the timezone for the upsert data must be UTC:
 
 ```Python
@@ -871,35 +967,36 @@ mkservice.upsertData(data)
 ```
 
 Upsert has optional switches that can be applied
+
 ```
 mkservice.upsertData(data, deferCommandExecution, deferDataGeneration, keepNulls, upsertMode)
 ```
+
 The switch details are,
 
 deferCommandExecution (true/false) choose between syncronoys and asyncronous command execution, default is false.
 deferDataGeneration (true/false) choose between syncronoys and asyncronous precomputed data generation, default is true.
 keepNulls (true/false) if true then nulls are written in the curve replacing any data present for the instant, default is false.
 upsertMode (Merge/Replace) for VersionedTimeSeries the merge writes in to the curve replacing existing data for an existing instant, replace writes the payload removing any previous data for the version. Leaving Null/None/Empty is equivalent to Merge.
 
-| DATETIME | EXISTING | PAYLOAD | MERGE | REPALACE |
-|---|---|---|---|---|
+| DATETIME     | EXISTING   | PAYLOAD    | MERGE      | REPLACE    |
+| ------------ | ---------- | ---------- | ---------- | ---------- |
 | VERSION NAME | 2025-01-01 | 2025-01-01 | 2025-01-01 | 2025-01-01 |
-| 2025-01-01 |        |        |        |        |
-| 2025-01-02 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-03 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-04 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-05 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-06 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-07 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-08 |        | 222.22 | 222.22 | 222.22 |
-| 2025-01-09 |        |        |        |        |
-| 2025-01-10 |        |        |        |        |
-| 2025-01-11 | 999.99 |        | 999.99 |        |
-| 2025-01-12 | 999.99 |        | 999.99 |        |
-| 2025-01-13 | 999.99 |        | 999.99 |        |
-| 2025-01-14 |        |        |        |        |
-| 2025-01-15 |        |        |        |        |
-
+| 2025-01-01   |            |            |            |            |
+| 2025-01-02   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-03   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-04   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-05   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-06   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-07   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-08   |            | 222.22     | 222.22     | 222.22     |
+| 2025-01-09   |            |            |            |            |
+| 2025-01-10   |            |            |            |            |
+| 2025-01-11   | 999.99     |            | 999.99     |            |
+| 2025-01-12   | 999.99     |            | 999.99     |            |
+| 2025-01-13   | 999.99     |            | 999.99     |            |
+| 2025-01-14   |            |            |            |            |
+| 2025-01-15   |            |            |            |            |
 
 ### Write Data in a Market Assessment Time Series
 
@@ -949,17 +1046,18 @@ mkservice.upsertData(marketAssessment)
 ```
 
 Upsert has optional switches that can be applied
+
 ```
 mkservice.upsertData(data, deferCommandExecution, deferDataGeneration, keepNulls, upsertMode)
 ```
+
 The switch details are,
 
 deferCommandExecution (true/false) choose between syncronoys and asyncronous command execution, default is false.
 deferDataGeneration (true/false) choose between syncronoys and asyncronous precomputed data generation, default is true.
 keepNulls (true/false) if true then nulls are written in the curve replacing any data present for the instant, default is false.
 upsertMode (Merge/Replace) for MarketAssessment merge adds the new products to the existing and overwrites existing with the new ones while replace replaces all the existing products with the new ones. Leaving Null/None/Empty is equivalent to Merge.
 
-
 ### Write Data in a Bid Ask Time Series
 
 ```Python
@@ -1007,17 +1105,18 @@ mkservice.upsertData(bidAsk)
 ```
 
 Upsert has optional switches that can be applied
+
 ```
 mkservice.upsertData(data, deferCommandExecution, deferDataGeneration, keepNulls, upsertMode)
 ```
+
 The switch details are,
 
 deferCommandExecution (true/false) choose between syncronoys and asyncronous command execution, default is false.
 deferDataGeneration (true/false) choose between syncronoys and asyncronous precomputed data generation, default is true.
 keepNulls (true/false) if true then nulls are written in the curve replacing any data present for the instant, default is false.
 upsertMode (Merge/Replace) for BidAsk merge adds the new products to the existing and overwrites existing with the new ones while replace replaces all the existing products with the new ones. Leaving Null/None/Empty is equivalent to Merge.
 
-
 ### Write Data in an Auction Time Series
 
 ```Python
@@ -1065,17 +1164,18 @@ auctionRows = MarketData.UpsertData(MarketData.MarketDataIdentifier('PROVIDER',
 ```
 
 Upsert has optional switches that can be applied
+
 ```
 mkservice.upsertData(data, deferCommandExecution, deferDataGeneration, keepNulls, upsertMode)
 ```
+
 The switch details are,
 
 deferCommandExecution (true/false) choose between syncronoys and asyncronous command execution, default is false.
 deferDataGeneration (true/false) choose between syncronoys and asyncronous precomputed data generation, default is true.
 keepNulls (true/false) if true then nulls are written in the curve replacing any data present for the instant, default is false.
 upsertMode (Merge/Replace) for Auction merge and replace are equivalent. Leaving Null/None/Empty is equivalent to Merge.
 
-
 ## Delete Data in Artesian
 
 Using the MarketDataService is possible to delete MarketData and its curves.

samples/TestActual.py

@@ -5,7 +5,7 @@
 
 qs = QueryService(cfg)
 
-# AbsoluteRange - TimeZone - MultiIds
+# AbsoluteRange - TimeZone - MultiIds 
 test1 = (
     qs.createActual()
     .forMarketData(

samples/TestDerivedCfgTransform.py

@@ -0,0 +1,101 @@
+from datetime import datetime
+import Artesian
+from Artesian.Granularity import Granularity
+from Artesian.MarketData._Dto.DerivedCfg import DerivedCfg
+from Artesian.MarketData._Enum.DerivedAlgorithm import DerivedAlgorithm
+from Artesian.MarketData._Enum.MarketDataType import MarketDataType
+import time
+
+cfg = Artesian.ArtesianConfig("https://arkive.artesian.cloud/tenantName/", "APIKey")
+
+mkdservice = Artesian.MarketData.MarketDataService(cfg)
+
+# curveOne
+actualCurveOne = Artesian.MarketData.MarketDataEntityInput(
+    "TestProviderNameDerivedTransform",
+    "CurveOne",
+    Granularity.Hour,
+    MarketDataType.ActualTimeSerie,
+    "UTC"
+)
+
+registeredCurveOne = mkdservice.readMarketDataRegistryByName(
+     actualCurveOne.providerName, actualCurveOne.marketDataName
+)
+if registeredCurveOne is None:
+    registeredCurveOne = mkdservice.registerMarketData(actualCurveOne)
+
+marketIdentifierCurveOne = Artesian.MarketData.MarketDataIdentifier(
+    actualCurveOne.providerName, actualCurveOne.marketDataName
+)
+
+# mkdservice.deleteMarketData(registeredCurveOne.marketDataId)
+
+data = Artesian.MarketData.UpsertData(
+    marketIdentifierCurveOne,
+    "UTC",
+    rows={datetime(2020, 1, 1, h): 10 for h in range(0, 8)},
+)
+
+mkdservice.upsertData(data)
+
+curveIds = [registeredCurveOne.marketDataId]
+
+# Create DerivedCfgTransform
+derivedCfg = DerivedCfg(
+    version=1,
+    derivedAlgorithm=DerivedAlgorithm.Transform,
+    transform="SELECT Time, (Value + 1) as Value FROM $table",
+    orderedReferencedMarketDataIds=curveIds,
+)
+
+actualCurveDerived = Artesian.MarketData.MarketDataEntityInput(
+    "TestProviderNameDerivedTransform",
+    "CurveDerived",
+    Granularity.Hour,
+    MarketDataType.ActualTimeSerie,
+    "UTC",
+    derivedCfg=derivedCfg,
+)
+
+registeredDerived = mkdservice.readMarketDataRegistryByName(
+    actualCurveDerived.providerName, actualCurveDerived.marketDataName
+)
+
+# mkdservice.deleteMarketData(registeredDerived.marketDataId)
+
+if registeredDerived is None:
+    registeredDerived = mkdservice.registerMarketData(actualCurveDerived)
+
+# check that derivedCfg is as expected
+assert (
+    registeredDerived.derivedCfg is not None
+    and registeredDerived.derivedCfg.derivedAlgorithm == DerivedAlgorithm.Transform
+), "Derived Algorithm is not the expected (Transform)"
+
+marketIdentifierDerived = Artesian.MarketData.MarketDataIdentifier(
+    actualCurveDerived.providerName, actualCurveDerived.marketDataName
+)
+
+time.sleep(2)
+
+# get the derived curve and check values are in according to the configuration
+query = Artesian.Query.QueryService(cfg)
+
+res = (
+    query.createActual()
+    .forMarketData([registeredDerived.marketDataId])
+    .inAbsoluteDateRange("2020-01-01", "2020-01-02")
+    .inTimeZone("UTC")
+    .inGranularity(Granularity.Hour)
+    .execute()
+)
+
+print(res)
+
+for i in range(0, 8):
+    assert res[i]['D'] == 11
+
+# Delete the curves completely
+mkdservice.deleteMarketData(registeredCurveOne.marketDataId)
+mkdservice.deleteMarketData(registeredDerived.marketDataId)