Add the parameter: metastore.drop-partition-ignore-nonexistent to ignore exceptions when a Hive partition does not exist.#7969
Conversation
…ore exceptions when a Hive partition does not exist.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
A few comments on this PR:
-
.gitignorechange: The.cursor-svg-check/entry looks unrelated to this PR — likely IDE-specific. Please remove it or put it into a separate commit. -
Design question: When
MetaExceptionis caught withignorePartitionNonexistent=true, the code callspartitionExistsInHms()which issues a second HMS RPC (getPartition). If HMS is unhealthy (which is likely the cause of the originalMetaException), this verification call may also fail — and the fallback in that case conservatively returnstrue, causing the original exception to be rethrown anyway. This is a reasonable safety choice, but it means the option won't help when HMS is in a bad state.Consider documenting this behavior (i.e., "the option only helps when the specific partition is genuinely absent, not when HMS is having connectivity issues").
-
Title format: The PR title doesn't follow the project convention of
[module] description. Should be something like:[hive] Add metastore.drop-partition-ignore-nonexistent option. -
Option naming: The Paimon project typically uses shorter, more consistent option names.
metastore.drop-partition-ignore-nonexistentis quite long. Have you consideredmetastore.drop-partition.ignore-not-existsto be more consistent with other Paimon options?
| ### Tantivy lib ### | ||
| *libtantivy_jni* | ||
|
|
||
| .cursor-svg-check/ |
There was a problem hiding this comment.
remove this file from the PR
| @@ -461,6 +463,20 @@ public void dropPartitions(Identifier identifier, List<Map<String, String>> part | |||
| false)); | |||
| } catch (NoSuchObjectException e) { | |||
| // do nothing if the partition not exists | |||
| } catch (MetaException e) { | |||
| if (ignorePartitionNonexistent | |||
| && !partitionExistsInHms(identifier, partitionValues)) { | |||
There was a problem hiding this comment.
I think the MetaException returned by HMS is very likely still remaining in the HMS.
Check the processing logic of HMS:
1.ms.getPartition() → OK
2.ms.getDatabase() → NPE，MetaException
3.ms.dropPartition()
There was a problem hiding this comment.
Therefore, I suggest that when this configuration is enabled, we ignore HMS errors, log a warning, and then delete the Paimon data.
|
close #7964 |
3 participants
Fixed a bug that caused the Partition Expire operation to fail due to a non-existent Hive partition.