diff --git a/docs/guides/canister-management/logs.md b/docs/guides/canister-management/logs.md index 547f819a..41a3b3ed 100644 --- a/docs/guides/canister-management/logs.md +++ b/docs/guides/canister-management/logs.md @@ -102,6 +102,8 @@ icp canister logs -e ic --since-index 100 --until-index 200 Timestamp and index filters cannot be combined with `--follow`. +Filtering also changes which records are dropped when the selected logs exceed the response size. An unfiltered read trims the oldest records, so it returns the most recent logs (the output ends with the newest record). A filtered read trims the newest records instead, so it returns the oldest records that match the filter (the output starts with the oldest matching record), which lets you page forward through a range by advancing the start of the filter. + To output logs as JSON for programmatic processing: ```bash @@ -178,6 +180,8 @@ The default log buffer size is 4096 bytes. When the buffer fills up, older log e icp canister settings update -e ic --log-memory-limit 2mib ``` +The limit must be either 0 (no memory for logs) or at least 4096 bytes: values between 1 and 4095 are rejected. + Supported suffixes: `kb` (1,000 bytes), `kib` (1,024 bytes), `mb` (1,000,000 bytes), `mib` (1,048,576 bytes). In `icp.yaml`: ```yaml diff --git a/docs/guides/canister-management/settings.mdx b/docs/guides/canister-management/settings.mdx index c6f49179..2a5e7702 100644 --- a/docs/guides/canister-management/settings.mdx +++ b/docs/guides/canister-management/settings.mdx @@ -197,14 +197,12 @@ settings: ### Log memory limit -{/* Needs human verification: log_memory_limit is exposed by icp-cli but is absent from the canonical ic.did: verify whether this is a management canister setting or an icp-cli layer setting */} - Maximum memory for storing canister logs. Oldest logs are purged when usage exceeds this value. | Property | Value | |----------|-------| | Type | Integer or string with suffix | -| Max | 2 MiB | +| Valid values | 0, or 4096 bytes up to 2 MiB (values between 1 and 4095 are rejected) | | Default | 4096 bytes | | icp.yaml key | `log_memory_limit` | diff --git a/docs/references/ic-interface-spec/abstract-behavior.md b/docs/references/ic-interface-spec/abstract-behavior.md index f5070a79..ac18a6eb 100644 --- a/docs/references/ic-interface-spec/abstract-behavior.md +++ b/docs/references/ic-interface-spec/abstract-behavior.md @@ -130,6 +130,7 @@ The [WebAssembly System API](./canister-interface.md#system-api) is relatively l memory_usage_canister_history : Nat; memory_usage_chunk_store : Nat; memory_usage_snapshots : Nat; + memory_usage_log_memory_store : Nat; freezing_threshold : Nat; subnet_id : Principal; subnet_size : Nat; @@ -523,6 +524,7 @@ S = { last_install_timestamp: CanisterId ↦ Timestamp; canister_history: CanisterId ↦ CanisterHistory; canister_log_visibility: CanisterId ↦ CanisterLogVisibility; + canister_log_memory_limit: CanisterId ↦ Nat; canister_snapshot_visibility: CanisterId ↦ CanisterSnapshotVisibility; canister_status_visibility: CanisterId ↦ CanisterStatusVisibility; canister_logs: CanisterId ↦ [CanisterLog]; @@ -569,6 +571,8 @@ freezing_limit(compute_allocation, memory_allocation, freezing_threshold, memory ``` The (unspecified) functions `memory_usage_wasm_state(wasm_state)`, `memory_usage_raw_module(raw_module)`, `memory_usage_canister_history(canister_history)`, `memory_usage_chunk_store(chunk_store)`, and `memory_usage_snapshots(snapshots)` determine the canister's memory usage in bytes consumed by its Wasm state, raw Wasm binary, canister history, chunk store, and snapshots, respectively. +The (unspecified) function `memory_usage_log_memory_store(log_memory_limit)` determines the canister's memory usage in bytes consumed by the store holding its canister logs. +This memory is reserved based on the value `log_memory_limit` in canister settings and thus does not depend on the canister logs actually stored (in particular, it is zero if and only if `log_memory_limit` is zero). The freezing limit of canister `A` in state `S` can be obtained as follows: ``` @@ -581,7 +585,8 @@ freezing_limit(S, A) = memory_usage_raw_module(S.canisters[A].raw_module) + memory_usage_canister_history(S.canister_history[A]) + memory_usage_chunk_store(S.chunk_store[A]) + - memory_usage_snapshots(S.snapshots[A]), + memory_usage_snapshots(S.snapshots[A]) + + memory_usage_log_memory_store(S.canister_log_memory_limit[A]), S.canister_subnet[A].subnet_size, ) ``` @@ -603,7 +608,7 @@ liquid_balance(S, A) = The reasoning behind this is that resource payments first drain the reserved balance and only when the reserved balance gets to zero, they start draining the main balance. -The amount of cycles that need to be reserved after operations that allocate resources is modeled with an unspecified function `cycles_to_reserve(S, CanisterId, compute_allocation, memory_allocation, snapshots, CanState)` that depends on the old IC state, the id of the canister, the new allocations of the canister, the snapshots of the canister, and the new state of the canister. +The amount of cycles that need to be reserved after operations that allocate resources is modeled with an unspecified function `cycles_to_reserve(S, CanisterId, compute_allocation, memory_allocation, log_memory_limit, snapshots, CanState)` that depends on the old IC state, the id of the canister, the new allocations of the canister, the new log memory limit of the canister, the snapshots of the canister, and the new state of the canister. #### Initial state @@ -636,6 +641,7 @@ The initial state of the IC is last_install_timestamp = (); canister_history = (); canister_log_visibility = (); + canister_log_memory_limit = (); canister_snapshot_visibility = (); canister_status_visibility = (); canister_logs = (); @@ -850,6 +856,7 @@ liquid_balance(S, E.content.canister_id) ≥ 0 memory_usage_canister_history = memory_usage_canister_history(S.canister_history[E.content.canister_id]); memory_usage_chunk_store = memory_usage_chunk_store(S.chunk_store[E.content.canister_id]); memory_usage_snapshots = memory_usage_snapshots(S.snapshots[E.content.canister_id]); + memory_usage_log_memory_store = memory_usage_log_memory_store(S.canister_log_memory_limit[E.content.canister_id]); freezing_threshold = S.freezing_threshold[E.content.canister_id]; subnet_id = S.canister_subnet[E.content.canister_id].subnet_id; subnet_size = S.canister_subnet[E.content.canister_id].subnet_size; @@ -1321,6 +1328,7 @@ Env = { memory_usage_canister_history = memory_usage_canister_history(S.canister_history[M.receiver]); memory_usage_chunk_store = memory_usage_chunk_store(S.chunk_store[M.receiver]); memory_usage_snapshots = memory_usage_snapshots(S.snapshots[M.receiver]); + memory_usage_log_memory_store = memory_usage_log_memory_store(S.canister_log_memory_limit[M.receiver]); freezing_threshold = S.freezing_threshold[M.receiver]; subnet_id = S.canister_subnet[M.receiver].subnet_id; subnet_size = S.canister_subnet[M.receiver].subnet_size; @@ -1381,7 +1389,7 @@ if res.cycles_accepted ≤ Available (res.cycles_used + ∑ [ MAX_CYCLES_PER_RESPONSE + call.transferred_cycles | call ∈ res.new_calls ]) ≤ (S.balances[M.receiver] + res.cycles_accepted + (if Is_response then MAX_CYCLES_PER_RESPONSE else MAX_CYCLES_PER_MESSAGE)) - Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.snapshots[A.canister_id], New_state) + Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.canister_log_memory_limit[A.canister_id], S.snapshots[A.canister_id], New_state) New_balance = (S.balances[M.receiver] + res.cycles_accepted + (if Is_response then MAX_CYCLES_PER_RESPONSE else MAX_CYCLES_PER_MESSAGE)) - (res.cycles_used + ∑ [ MAX_CYCLES_PER_RESPONSE + call.transferred_cycles | call ∈ res.new_calls ]) @@ -1395,7 +1403,8 @@ if memory_usage_raw_module(S.canisters[M.receiver].raw_module) + memory_usage_canister_history(S.canister_history[M.receiver]) + memory_usage_chunk_store(S.chunk_store[M.receiver]) + - memory_usage_snapshots(S.snapshots[M.receiver]), + memory_usage_snapshots(S.snapshots[M.receiver]) + + memory_usage_log_memory_store(S.canister_log_memory_limit[M.receiver]), S.canister_subnet[M.receiver].subnet_size, ) New_reserved_balance ≤ S.reserved_balance_limits[M.receiver] @@ -1734,8 +1743,12 @@ if A.settings.environment_variables is not null: New_environment_variables = A.settings.environment_variables else: New_environment_variables = [] +if A.settings.log_memory_limit is not null: + New_canister_log_memory_limit = A.settings.log_memory_limit +else: + New_canister_log_memory_limit = 4096 -Cycles_reserved = cycles_to_reserve(S, Canister_id, New_compute_allocation, New_memory_allocation, null, EmptyCanister.wasm_state) +Cycles_reserved = cycles_to_reserve(S, Canister_id, New_compute_allocation, New_memory_allocation, New_canister_log_memory_limit, null, EmptyCanister.wasm_state) New_balance = M.transferred_cycles - Cycles_reserved New_reserved_balance = Cycles_reserved New_reserved_balance <= New_reserved_balance_limit @@ -1801,6 +1814,7 @@ S' = S with query_stats[Canister_id] = [] canister_history[Canister_id] = New_canister_history canister_log_visibility[Canister_id] = New_canister_log_visibility + canister_log_memory_limit[Canister_id] = New_canister_log_memory_limit canister_snapshot_visibility[Canister_id] = New_canister_snapshot_visibility canister_status_visibility[Canister_id] = New_canister_status_visibility canister_logs[Canister_id] = [] @@ -1839,6 +1853,9 @@ To avoid clashes with potential user ids or is derived from users or canisters, Only the controllers of the given canister can update the canister settings. +Changing the value `log_memory_limit` resizes the store holding the canister logs, which consumes cycles. +This cost is modeled by the (unspecified) variable `Cycles_used` that is zero unless the store is resized. + Conditions ```html @@ -1892,12 +1909,16 @@ if A.settings.environment_variables is not null: New_environment_variables = A.settings.environment_variables else: New_environment_variables = S.environment_variables[A.canister_id] +if A.settings.log_memory_limit is not null: + New_canister_log_memory_limit = A.settings.log_memory_limit +else: + New_canister_log_memory_limit = S.canister_log_memory_limit[A.canister_id] -Cycles_reserved = cycles_to_reserve(S, A.canister_id, New_compute_allocation, New_memory_allocation, S.snapshots[A.canister_id], S.canisters[A.canister_id].wasm_state) -New_balance = S.balances[A.canister_id] - Cycles_reserved +Cycles_reserved = cycles_to_reserve(S, A.canister_id, New_compute_allocation, New_memory_allocation, New_canister_log_memory_limit, S.snapshots[A.canister_id], S.canisters[A.canister_id].wasm_state) +New_balance = S.balances[A.canister_id] - Cycles_used - Cycles_reserved New_reserved_balance = S.reserved_balances[A.canister_id] + Cycles_reserved New_reserved_balance ≤ New_reserved_balance_limit -if New_compute_allocation > S.compute_allocation[A.canister_id] or New_memory_allocation > S.memory_allocation[A.canister_id] or Cycles_reserved > 0: +if New_compute_allocation > S.compute_allocation[A.canister_id] or New_memory_allocation > S.memory_allocation[A.canister_id] or New_canister_log_memory_limit > S.canister_log_memory_limit[A.canister_id] or Cycles_used > 0 or Cycles_reserved > 0: liquid_balance(S', A.canister_id) ≥ 0 S.canister_history[A.canister_id] = { @@ -1943,6 +1964,7 @@ S' = S with canister_version[A.canister_id] = S.canister_version[A.canister_id] + 1 if A.settings.log_visibility is not null: canister_log_visibility[A.canister_id] = A.settings.log_visibility + canister_log_memory_limit[A.canister_id] = New_canister_log_memory_limit if A.settings.snapshot_visibility is not null: canister_snapshot_visibility[A.canister_id] = A.settings.snapshot_visibility if A.settings.status_visibility is not null: @@ -1976,6 +1998,7 @@ canister_status(S, Canister_id) = reserved_cycles_limit = S.reserved_balance_limit[Canister_id]; minimum_incoming_canister_call_cycles = S.minimum_incoming_canister_call_cycles[Canister_id]; log_visibility = S.canister_log_visibility[Canister_id]; + log_memory_limit = S.canister_log_memory_limit[Canister_id]; snapshot_visibility = S.canister_snapshot_visibility[Canister_id]; status_visibility = S.canister_status_visibility[Canister_id]; wasm_memory_limit = S.wasm_memory_limit[Canister_id]; @@ -1997,7 +2020,8 @@ canister_status(S, Canister_id) = memory_usage_raw_module(S.canisters[Canister_id].raw_module) + memory_usage_canister_history(S.canister_history[Canister_id]) + memory_usage_chunk_store(S.chunk_store[Canister_id]) + - memory_usage_snapshots(S.snapshots[Canister_id]), + memory_usage_snapshots(S.snapshots[Canister_id]) + + memory_usage_log_memory_store(S.canister_log_memory_limit[Canister_id]), S.freezing_threshold[Canister_id], S.canister_subnet[Canister_id].subnet_size, ); @@ -2386,6 +2410,7 @@ Env = { memory_usage_canister_history = memory_usage_canister_history(New_canister_history); memory_usage_chunk_store = memory_usage_chunk_store(New_chunk_store); memory_usage_snapshots = memory_usage_snapshots(S.snapshots[A.canister_id]); + memory_usage_log_memory_store = memory_usage_log_memory_store(S.canister_log_memory_limit[A.canister_id]); freezing_threshold = S.freezing_threshold[A.canister_id]; subnet_id = S.canister_subnet[A.canister_id].subnet_id; subnet_size = S.canister_subnet[A.canister_id].subnet_size; @@ -2394,7 +2419,7 @@ Env = { canister_version = S.canister_version[A.canister_id] + 1; } Mod.init(A.canister_id, A.arg, M.caller, Env) = Return {new_state = New_state; new_certified_data = New_certified_data; new_global_timer = New_global_timer; cycles_used = Cycles_used;} -Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.snapshots[A.canister_id], New_state) +Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.canister_log_memory_limit[A.canister_id], S.snapshots[A.canister_id], New_state) New_balance = S.balances[A.canister_id] - Cycles_used - Cycles_reserved New_reserved_balance = S.reserved_balances[A.canister_id] + Cycles_reserved New_reserved_balance ≤ S.reserved_balance_limits[A.canister_id] @@ -2498,6 +2523,7 @@ Env = { memory_usage_canister_history = memory_usage_canister_history(S.canister_history[A.canister_id]); memory_usage_chunk_store = memory_usage_chunk_store(S.chunk_store[A.canister_id]); memory_usage_snapshots = memory_usage_snapshots(S.snapshots[A.canister_id]); + memory_usage_log_memory_store = memory_usage_log_memory_store(S.canister_log_memory_limit[A.canister_id]); freezing_threshold = S.freezing_threshold[A.canister_id]; subnet_id = S.canister_subnet[A.canister_id].subnet_id; subnet_size = S.canister_subnet[A.canister_id].subnet_size; @@ -2559,7 +2585,7 @@ Env2 = Env with { Mod.post_upgrade(Persisted_state, A.arg, M.caller, Env2) = Return {new_state = New_state; new_certified_data = New_certified_data'; new_global_timer = New_global_timer; cycles_used = Cycles_used';} -Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.snapshots[A.canister_id], New_state) +Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.canister_log_memory_limit[A.canister_id], S.snapshots[A.canister_id], New_state) New_balance = S.balances[A.canister_id] - Cycles_used - Cycles_used' - Cycles_reserved New_reserved_balance = S.reserved_balances[A.canister_id] + Cycles_reserved New_reserved_balance ≤ S.reserved_balance_limits[A.canister_id] @@ -2994,7 +3020,8 @@ S with certified_data[A.canister_id] = (deleted) canister_history[A.canister_id] = (deleted) canister_log_visibility[A.canister_id] = (deleted) - canister_snapshot_visibility[A.canister_id] = (deleted) + canister_log_memory_limit[A.canister_id] = (deleted) + canister_snapshot_visibility[A.canister_id] = (deleted) canister_status_visibility[A.canister_id] = (deleted) canister_logs[A.canister_id] = (deleted) query_stats[A.canister_id] = (deleted) @@ -3199,9 +3226,13 @@ if A.settings.environment_variables is not null: New_environment_variables = A.settings.environment_variables else: New_environment_variables = [] +if A.settings.log_memory_limit is not null: + New_canister_log_memory_limit = A.settings.log_memory_limit +else: + New_canister_log_memory_limit = 4096 -Cycles_reserved = cycles_to_reserve(S, Canister_id, New_compute_allocation, New_memory_allocation, null, EmptyCanister.wasm_state) +Cycles_reserved = cycles_to_reserve(S, Canister_id, New_compute_allocation, New_memory_allocation, New_canister_log_memory_limit, null, EmptyCanister.wasm_state) if A.amount is not null: New_balance = A.amount - Cycles_reserved else: @@ -3268,6 +3299,7 @@ S' = S with certified_data[Canister_id] = "" canister_history[Canister_id] = New_canister_history canister_log_visibility[Canister_id] = New_canister_log_visibility + canister_log_memory_limit[Canister_id] = New_canister_log_memory_limit canister_snapshot_visibility[Canister_id] = New_canister_snapshot_visibility canister_status_visibility[Canister_id] = New_canister_status_visibility canister_logs[Canister_id] = [] @@ -3351,7 +3383,7 @@ New_snapshot = Snapshot { New_snapshots = S.snapshots[A.canister_id] with A.replace_snapshot = (undefined) Snapshot_id = New_snapshot -Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], New_snapshots, S.canisters[A.canister_id]) +Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.canister_log_memory_limit[A.canister_id], New_snapshots, S.canisters[A.canister_id]) New_balance = S.balances[A.canister_id] - Cycles_used - Cycles_reserved New_reserved_balance = S.reserved_balances[A.canister_id] + Cycles_reserved New_reserved_balance ≤ S.reserved_balance_limits[A.canister_id] @@ -3416,7 +3448,7 @@ New_snapshot = Snapshot { New_snapshots = S.snapshots[A.canister_id] with A.replace_snapshot = (undefined) Snapshot_id = New_snapshot -Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], New_snapshots, EmptyCanister) +Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.canister_log_memory_limit[A.canister_id], New_snapshots, EmptyCanister) New_balance = S.balances[A.canister_id] - Cycles_used - Cycles_reserved New_reserved_balance = S.reserved_balances[A.canister_id] + Cycles_reserved New_reserved_balance ≤ S.reserved_balance_limits[A.canister_id] @@ -3542,7 +3574,7 @@ if Snapshot.source = MetadataUpload and Snapshot.on_low_wasm_memory_hook_status else: New_on_low_wasm_memory_hook_status = S.on_low_wasm_memory_hook_status[A.canister_id] -Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.snapshots[A.canister_id], New_state) +Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.canister_log_memory_limit[A.canister_id], S.snapshots[A.canister_id], New_state) New_balance = S.balances[A.canister_id] - Cycles_used - Cycles_reserved New_reserved_balance = S.reserved_balances[A.canister_id] + Cycles_reserved New_reserved_balance ≤ S.reserved_balance_limits[A.canister_id] @@ -3741,7 +3773,7 @@ New_snapshot = Snapshot { New_snapshots = S.snapshots[A.canister_id] with A.replace_snapshot = (undefined) Snapshot_id = New_snapshot -Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], New_snapshots, S.canisters[A.canister_id]) +Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.canister_log_memory_limit[A.canister_id], New_snapshots, S.canisters[A.canister_id]) New_balance = S.balances[A.canister_id] - Cycles_used - Cycles_reserved New_reserved_balance = S.reserved_balances[A.canister_id] + Cycles_reserved New_reserved_balance ≤ S.reserved_balance_limits[A.canister_id] @@ -3816,7 +3848,7 @@ else if A.kind = WasmChunk: New_snapshots = S.snapshots[A.canister_id] with Snapshot_id = New_snapshot -Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], New_snapshots, S.canisters[A.canister_id]) +Cycles_reserved = cycles_to_reserve(S, A.canister_id, S.compute_allocation[A.canister_id], S.memory_allocation[A.canister_id], S.canister_log_memory_limit[A.canister_id], New_snapshots, S.canisters[A.canister_id]) New_balance = S.balances[A.canister_id] - Cycles_used - Cycles_reserved New_reserved_balance = S.reserved_balances[A.canister_id] + Cycles_reserved New_reserved_balance ≤ S.reserved_balance_limits[A.canister_id] @@ -3912,6 +3944,113 @@ S with ``` +#### IC Management Canister: Canister logs {#ic-mgmt-canister-fetch-canister-logs} + +Given a state `S`, `Canister_id`, `filter`, and `Sender`, we define + +```html + +canister_logs(S, Canister_id, filter) = + let Selected_logs = the sublist of S.canister_logs[Canister_id] (preserving order) such that + if filter = by_idx Range: + Log ∈ Selected_logs ⟺ Log ∈ S.canister_logs[Canister_id] ∧ Range.start <= Log.idx ∧ Log.idx < Range.end + else if filter = by_timestamp_nanos Range: + Log ∈ Selected_logs ⟺ Log ∈ S.canister_logs[Canister_id] ∧ Range.start <= Log.timestamp_nanos ∧ Log.timestamp_nanos < Range.end + else: + Log ∈ Selected_logs ⟺ Log ∈ S.canister_logs[Canister_id] + in + if filter = null: + the longest suffix Younger_logs of Selected_logs + such that canister_log_memory_usage(Younger_logs) ≤ max_response_size + else: + the longest prefix Older_logs of Selected_logs + such that canister_log_memory_usage(Older_logs) ≤ max_response_size +max_response_size = +is_sender_authorized(S, Canister_id, Sender) = + (S[Canister_id].canister_log_visibility = Public) + or + (S[Canister_id].canister_log_visibility = Controllers and Sender in S[Canister_id].controllers) + or + (S[Canister_id].canister_log_visibility = AllowedViewers Principals and (Sender in S[Canister_id].controllers or Sender in Principals)) + +``` + +When the selected logs do not all fit within a single response, they are trimmed to fit and the direction of trimming depends on whether a filter is provided. +An unfiltered read trims the oldest log records (keeps the longest suffix), so the response ends with the newest log record. +A filtered read trims the newest log records (keeps the longest prefix), so the response starts with the oldest log record satisfying the filter. +Thus an unfiltered read surfaces the most recent activity, while a filtered read can page forward through logs starting from the beginning of the requested range. + +Conditions + +```html + +S.messages = Older_messages · CallMessage M · Younger_messages +(M.queue = Unordered) or (∀ CallMessage M' | FuncMessage M' ∈ Older_messages. M'.queue ≠ M.queue) +M.callee = ic_principal +M.method_name = 'fetch_canister_logs' +M.arg = candid(A) +is_sender_authorized(S, A.canister_id, M.caller) + +``` + +State after + +```html + +S with + messages = Older_messages · Younger_messages · + ResponseMessage { + origin = M.origin + response = candid(canister_logs(S, A.canister_id, A.filter)) + refunded_cycles = M.transferred_cycles + } + +``` + +The IC method `fetch_canister_logs` can also be invoked via management canister query calls. +They are calls to `/api/v3/canister//query` +with CBOR content `Q` such that `Q.canister_id = ic_principal`. + +Submitted request to `/api/v3/canister//query` + +```html + +E : Envelope + +``` + +Conditions + +```html + +E.content = CanisterQuery Q +Q.canister_id = ic_principal +Q.method_name = 'fetch_canister_logs' +|Q.nonce| <= 32 +is_effective_canister_id(E.content, ECID) +S.system_time <= Q.ingress_expiry or Q.sender = anonymous_id +Q.arg = candid(A) +A.canister_id ∈ verify_envelope(E, Q.sender, S.system_time) +is_sender_authorized(S, A.canister_id, Q.sender) + +``` + +Query response `R`: + +```html + +{status: "replied"; reply: {arg: candid(canister_logs(S, A.canister_id, A.filter))}, signatures: Sigs} + +``` + +where the query `Q`, the response `R`, and a certificate `Cert` that is obtained by requesting the path `/subnet` in a **separate** read state request to `/api/v3/canister//read_state` satisfy the following: + +```html + +verify_response(Q, R, Cert) ∧ lookup(["time"], Cert) = Found S.system_time // or "recent enough" + +``` + #### Callback invocation When an inter-canister call has been responded to, we can queue the call to the callback. @@ -4130,6 +4269,7 @@ S with global_timer[CanisterId] = 0 compute_allocation[Canister_id] = 0 memory_allocation[Canister_id] = 0 + canister_log_memory_limit[Canister_id] = 0 messages = S.messages · [ ResponseMessage { @@ -4257,6 +4397,8 @@ S with canister_history[Canister_id] = (deleted) canister_log_visibility[New_canister_id] = S.canister_log_visibility[Canister_id] canister_log_visibility[Canister_id] = (deleted) + canister_log_memory_limit[New_canister_id] = S.canister_log_memory_limit[Canister_id] + canister_log_memory_limit[Canister_id] = (deleted) canister_snapshot_visibility[New_canister_id] = S.canister_snapshot_visibility[Canister_id] canister_snapshot_visibility[Canister_id] = (deleted) canister_status_visibility[New_canister_id] = S.canister_status_visibility[Canister_id] @@ -4422,16 +4564,19 @@ S with ``` -#### Trimming canister logs +#### Purging canister logs -Canister logs can be trimmed if their total length exceeds 4KiB. +Oldest canister logs are purged if the total memory used for canister logs exceeds the value `log_memory_limit` in canister settings. + +The (unspecified) function `canister_log_memory_usage(logs)` models the total memory used by `logs`. Conditions ```html S.canister_logs[CanisterId] = Older_logs · Newer_logs -SUM { |l| | l <- Older_logs } > 4KiB +canister_log_memory_usage(Older_logs · Newer_logs) > S.canister_log_memory_limit[CanisterId] +canister_log_memory_usage(Newer_logs) ≤ S.canister_log_memory_limit[CanisterId] ``` @@ -4444,60 +4589,6 @@ S with ``` -#### IC Management Canister: Canister logs (query call) {#ic-mgmt-canister-fetch-canister-logs} - -This section specifies management canister query calls. -They are calls to `/api/v3/canister//query` -with CBOR content `Q` such that `Q.canister_id = ic_principal`. - -The management canister offers the method `fetch_canister_logs` -that can be called as a query call and -returns logs of a requested canister. - -Submitted request to `/api/v3/canister//query` - -```html - -E : Envelope - -``` - -Conditions - -```html - -E.content = CanisterQuery Q -Q.canister_id = ic_principal -Q.method_name = 'fetch_canister_logs' -|Q.nonce| <= 32 -is_effective_canister_id(E.content, ECID) -S.system_time <= Q.ingress_expiry or Q.sender = anonymous_id -Q.arg = candid(A) -A.canister_id ∈ verify_envelope(E, Q.sender, S.system_time) -(S[A.canister_id].canister_log_visibility = Public) - or - (S[A.canister_id].canister_log_visibility = Controllers and Q.sender in S[A.canister_id].controllers) - or - (S[A.canister_id].canister_log_visibility = AllowedViewers Principals and (Q.sender in S[A.canister_id].controllers or Q.sender in Principals)) - -``` - -Query response `R`: - -```html - -{status: "replied"; reply: {arg: candid(S.canister_logs[A.canister_id])}, signatures: Sigs} - -``` - -where the query `Q`, the response `R`, and a certificate `Cert` that is obtained by requesting the path `/subnet` in a **separate** read state request to `/api/v3/canister//read_state` satisfy the following: - -```html - -verify_response(Q, R, Cert) ∧ lookup(["time"], Cert) = Found S.system_time // or "recent enough" - -``` - #### IC Management Canister: List canisters (query call) {#ic-mgmt-canister-list-canisters} This section specifies the `list_canisters` management canister query call. @@ -4611,13 +4702,9 @@ management_canister_query(S, Caller, Method_name, Arg) = Return (Reply (candid()), Cycles_used) if Method_name = 'fetch_canister_logs' and Arg = candid(A) and S.canister_subnet[A.canister_id].subnet_id = S.canister_subnet[Caller].subnet_id and - ((S.canister_log_visibility[A.canister_id] = Public) - or - (S.canister_log_visibility[A.canister_id] = Controllers and Caller ∈ S.controllers[A.canister_id]) - or - (S.canister_log_visibility[A.canister_id] = AllowedViewers Principals and (Caller ∈ S.controllers[A.canister_id] or Caller ∈ Principals))) + is_sender_authorized(S, A.canister_id, Caller) then - Return (Reply (candid(S.canister_logs[A.canister_id])), Cycles_used) + Return (Reply (candid(canister_logs(S, A.canister_id, A.filter))), Cycles_used) if Method_name = 'list_canisters' and Caller ∈ S.subnet_admins[S.canister_subnet[Caller]] then @@ -4650,6 +4737,7 @@ composite_query_helper(S, Cycles, Depth, Root_canister_id, Caller, Caller_info_d memory_usage_canister_history = memory_usage_canister_history(S.canister_history[Canister_id]); memory_usage_chunk_store = memory_usage_chunk_store(S.chunk_store[Canister_id]); memory_usage_snapshots = memory_usage_snapshots(S.snapshots[Canister_id]); + memory_usage_log_memory_store = memory_usage_log_memory_store(S.canister_log_memory_limit[Canister_id]); freezing_threshold = S.freezing_threshold[Canister_id]; subnet_id = S.canister_subnet[Canister_id].subnet_id; subnet_size = S.canister_subnet[Canister_id].subnet_size; @@ -5018,7 +5106,8 @@ liquid_balance(es) = es.params.sysenv.memory_usage_raw_module + es.params.sysenv.memory_usage_canister_history + es.params.sysenv.memory_usage_chunk_store + - es.params.sysenv.memory_usage_snapshots, + es.params.sysenv.memory_usage_snapshots + + es.params.sysenv.memory_usage_log_memory_store, es.params.sysenv.subnet_size, ) ) diff --git a/docs/references/ic-interface-spec/management-canister.md b/docs/references/ic-interface-spec/management-canister.md index 7fa8980f..a137b263 100644 --- a/docs/references/ic-interface-spec/management-canister.md +++ b/docs/references/ic-interface-spec/management-canister.md @@ -105,6 +105,14 @@ The optional `settings` parameter can be used to set the following settings: Default value: `controllers`. +- `log_memory_limit` (`nat`) + + Must be either `0` or a number between `4096` and `2097152` (`2 MiB`), inclusively, and indicates the maximum amount of memory used for canister logs. + In particular, values between `1` and `4095`, inclusively, are not allowed. + Oldest canister logs are purged if the total memory used for canister logs exceeds this value. + + Default value: `4096`. + - `snapshot_visibility` (`snapshot_visibility`) Controls who can access the canister's snapshots through the following endpoints of the management canister: @@ -332,6 +340,8 @@ Regardless of this setting, the canister itself and subnet admins can always req * `snapshots_size`: Represents the memory consumed by all snapshots that belong to this canister. + * `log_memory_store_size`: Represents the memory used by canister logs of the canister. + All sizes are expressed in bytes. ### IC method `canister_metrics` {#ic-canister_metrics} @@ -950,13 +960,14 @@ A snapshot may be deleted only by the controllers of the canister that the snaps ### IC method `fetch_canister_logs` {#ic-fetch_canister_logs} -This method can only be called via non-replicated (query) calls: by external users directly and by canisters from composite query methods and their callbacks, i.e., it cannot be called via replicated calls. +This method can be called by canisters via replicated calls, i.e., it cannot be called by external users via replicated (update) calls. +This method can also be called via non-replicated (query) calls: by external users directly and by canisters from composite query methods and their callbacks. A call from a composite query is executed against the state of the subnet hosting the calling canister and can thus only target canisters hosted by that subnet. Given a canister ID as input, this method returns a vector of logs of that canister including its trap messages. The canister logs are *not* collected in canister methods running in non-replicated mode (NRQ, TQ, CQ, CRy, CRt, CC, and F modes, as defined in [Overview of imports](./canister-interface.md#system-api-imports)) and the canister logs are *purged* when the canister is reinstalled or uninstalled. -The total size of all returned logs does not exceed 4KiB. -If new logs are added resulting in exceeding the maximum total log size of 4KiB, the oldest logs will be removed. +The total size of all returned logs does not exceed an implementation-defined constant chosen so as not to exceed the maximum response size. +Oldest canister logs are purged if the total memory used for canister logs exceeds the value `log_memory_limit` in canister settings. Logs persist across canister upgrades and they are deleted if the canister is reinstalled or uninstalled. The log visibility is defined in the `log_visibility` field of `canister_settings` and can be one of the following variants: @@ -971,6 +982,14 @@ A single log is a record with the following fields: - `timestamp_nanos` (`nat64`): the timestamp as nanoseconds since 1970-01-01 at which the log was recorded; - `content` (`blob`): the actual content of the log; +To filter canister logs, an optional filter can be provided and have one of the following variants: +- `by_idx` (`record { start : nat64; end : nat64 }`): only logs are returned whose `idx` is within the provided range (`start` is inclusive, but `end` is exclusive); +- `by_timestamp_nanos` (`record { start : nat64; end : nat64 }`): only logs are returned whose `timestamp_nanos` is within the provided range (`start` is inclusive, but `end` is exclusive). + +When the logs selected for the response do not all fit within a single response, they are trimmed to fit, and the direction of trimming differs between filtered and unfiltered reads: +- An **unfiltered** read trims the **oldest** log records, so the response ends with the newest log record. This surfaces the most recent activity. +- A **filtered** read trims the **newest** log records, so the response starts with the oldest log record satisfying the filter. This lets a filtered read page forward through logs starting from the beginning of the requested range. + :::warning The response of a query comes from a single replica, and is therefore not appropriate for security-sensitive applications. diff --git a/public/references/ic.did b/public/references/ic.did index 23c9e10b..1f9d8cf9 100644 --- a/public/references/ic.did +++ b/public/references/ic.did @@ -33,6 +33,7 @@ type canister_settings = record { reserved_cycles_limit : opt nat; minimum_incoming_canister_call_cycles : opt nat; log_visibility : opt log_visibility; + log_memory_limit : opt nat; snapshot_visibility : opt snapshot_visibility; status_visibility : opt status_visibility; wasm_memory_limit : opt nat; @@ -48,6 +49,7 @@ type definite_canister_settings = record { reserved_cycles_limit : nat; minimum_incoming_canister_call_cycles : nat; log_visibility : log_visibility; + log_memory_limit : nat; snapshot_visibility : snapshot_visibility; status_visibility : status_visibility; wasm_memory_limit : nat; @@ -310,6 +312,7 @@ type canister_status_result = record { canister_history_size : nat; wasm_chunk_store_size : nat; snapshots_size : nat; + log_memory_store_size : nat; }; cycles : nat; reserved_cycles : nat; @@ -509,6 +512,10 @@ type delete_canister_snapshot_args = record { type fetch_canister_logs_args = record { canister_id : canister_id; + filter : opt variant { + by_idx : record { start : nat64; end : nat64 }; + by_timestamp_nanos : record { start : nat64; end : nat64 }; + } }; type canister_log_record = record {