Skip to content

PR Validation Flaky Test #6684

New issue
New issue
Open
Open
PR Validation Flaky Test#6684
Labels
area-TrainersMachine learning trainers and training APIsgreendevelopmenttest-bugProblem in test source code (most likely)
Milestone
ML.NET Future

Description

@michaelgsharp
michaelgsharp
opened on May 15, 2023

Microsoft.ML.RunTests.TestPredictors.MulticlassLRTest is flaky on arm/arm64 devices. Disabling the test for now on non x86/x64 devices until we figure out the issue.

Microsoft.ML.RunTests.TestPredictors.MulticlassLRTest [FAIL]
�[31;1m�[m�[37m      Assert.Equal() Failure
�[m�[37m      Expected: 0
�[m�[37m      Actual:   2
�[m�[30;1m      Stack Trace:
�[m�[37m        /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs(201,0): at Microsoft.ML.RunTests.BaseTestBaseline.Done()
�[m�[37m        /__w/1/s/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs(156,0): at Microsoft.ML.RunTests.TestPredictors.MulticlassLRTest()
�[m�[30;1m      Output:
�[m�[37m        Running 'MulticlassLogisticRegression' on 'iris'
�[m�[37m          Running as: TrainTest tr=MulticlassLogisticRegression{l1=0.001 l2=0.1 ot=1e-3 nt=1} data=/root/helix/work/correlation/test/data/iris.txt seed=1 test=/root/helix/work/correlation/test/data/iris.txt xf=Term{col=Label} out={/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-model.zip} dout={/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris.txt} norm=no
�[m�[37m        maml.exe TrainTest test=/root/helix/work/correlation/test/data/iris.txt tr=MulticlassLogisticRegression{l1=0.001 l2=0.1 ot=1e-3 nt=1} norm=No dout=/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris.txt data=/root/helix/work/correlation/test/data/iris.txt out=/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-model.zip seed=1 xf=Term{col=Label}
�[m�[37m        
�[m�[37m        
�[m�[37m        Not adding a normalizer.
�[m�[37m        
�[m�[37m        
�[m�[37m        Beginning optimization
�[m�[37m        
�[m�[37m        
�[m�[37m        num vars: 15
�[m�[37m        
�[m�[37m        
�[m�[37m        improvement criterion: Mean Improvement
�[m�[37m        
�[m�[37m        
�[m�[37m        L1 regularization selected 15 of 15 weights.
�[m�[37m        
�[m�[37m        
�[m�[37m        Not training a calibrator because it is not needed.
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        Confusion table
�[m�[37m        
�[m�[37m        
�[m�[37m                  ||========================
�[m�[37m        
�[m�[37m        
�[m�[37m        PREDICTED ||     0 |     1 |     2 | Recall
�[m�[37m        
�[m�[37m        
�[m�[37m        TRUTH     ||========================
�[m�[37m        
�[m�[37m        
�[m�[37m                0 ||    50 |     0 |     0 | 1.0000
�[m�[37m        
�[m�[37m        
�[m�[37m                1 ||     0 |    48 |     2 | 0.9600
�[m�[37m        
�[m�[37m        
�[m�[37m                2 ||     0 |     1 |    49 | 0.9800
�[m�[37m        
�[m�[37m        
�[m�[37m                  ||========================
�[m�[37m        
�[m�[37m        
�[m�[37m        Precision ||1.0000 |0.9796 |0.9608 |
�[m�[37m        
�[m�[37m        
�[m�[37m        Accuracy(micro-avg): 0.980000
�[m�[37m        
�[m�[37m        
�[m�[37m        Accuracy(macro-avg): 0.980000
�[m�[37m        
�[m�[37m        
�[m�[37m        Log-loss:           0.072171
�[m�[37m        
�[m�[37m        
�[m�[37m        Log-loss reduction: 0.934307
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        OVERALL RESULTS
�[m�[37m        
�[m�[37m        
�[m�[37m        ---------------------------------------
�[m�[37m        
�[m�[37m        
�[m�[37m        Accuracy(micro-avg): 0.980000 (0.0000)
�[m�[37m        
�[m�[37m        
�[m�[37m        Accuracy(macro-avg): 0.980000 (0.0000)
�[m�[37m        
�[m�[37m        
�[m�[37m        Log-loss:           0.072171 (0.0000)
�[m�[37m        
�[m�[37m        
�[m�[37m        Log-loss reduction: 0.934307 (0.0000)
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        ---------------------------------------
�[m�[37m        
�[m�[37m        
�[m�[37m        Physical memory usage(MB): 120
�[m�[37m        
�[m�[37m        
�[m�[37m        Virtual memory usage(MB): 3549
�[m�[37m        
�[m�[37m        
�[m�[37m        5/15/2023 5:29:00 PM	 Time elapsed(s): 0.028
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        Comparing /root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-out.txt and /root/helix/work/correlation/test/BaselineOutput/Common/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-out.txt
�[m�[37m        Output matches baseline: 'MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-out.txt'
�[m�[37m        Comparing /root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris.txt and /root/helix/work/correlation/test/BaselineOutput/Common/MulticlassLogisticRegression/netcoreapp/MulticlassLogisticRegression-TrainTest-iris.txt
�[m�[37m        Output matches baseline: 'MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris.txt'
�[m�[37m        maml.exe Test dout=/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris.txt data=/root/helix/work/correlation/test/data/iris.txt in=/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-model.zip seed=1
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        Confusion table
�[m�[37m        
�[m�[37m        
�[m�[37m                  ||========================
�[m�[37m        
�[m�[37m        
�[m�[37m        PREDICTED ||     0 |     1 |     2 | Recall
�[m�[37m        
�[m�[37m        
�[m�[37m        TRUTH     ||========================
�[m�[37m        
�[m�[37m        
�[m�[37m                0 ||    50 |     0 |     0 | 1.0000
�[m�[37m        
�[m�[37m        
�[m�[37m                1 ||     0 |    48 |     2 | 0.9600
�[m�[37m        
�[m�[37m        
�[m�[37m                2 ||     0 |     1 |    49 | 0.9800
�[m�[37m        
�[m�[37m        
�[m�[37m                  ||========================
�[m�[37m        
�[m�[37m        
�[m�[37m        Precision ||1.0000 |0.9796 |0.9608 |
�[m�[37m        
�[m�[37m        
�[m�[37m        Accuracy(micro-avg): 0.980000
�[m�[37m        
�[m�[37m        
�[m�[37m        Accuracy(macro-avg): 0.980000
�[m�[37m        
�[m�[37m        
�[m�[37m        Log-loss:           0.072171
�[m�[37m        
�[m�[37m        
�[m�[37m        Log-loss reduction: 0.934307
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        OVERALL RESULTS
�[m�[37m        
�[m�[37m        
�[m�[37m        ---------------------------------------
�[m�[37m        
�[m�[37m        
�[m�[37m        Accuracy(micro-avg): 0.980000 (0.0000)
�[m�[37m        
�[m�[37m        
�[m�[37m        Accuracy(macro-avg): 0.980000 (0.0000)
�[m�[37m        
�[m�[37m        
�[m�[37m        Log-loss:           0.072171 (0.0000)
�[m�[37m        
�[m�[37m        
�[m�[37m        Log-loss reduction: 0.934307 (0.0000)
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        ---------------------------------------
�[m�[37m        
�[m�[37m        
�[m�[37m        Physical memory usage(MB): 120
�[m�[37m        
�[m�[37m        
�[m�[37m        Virtual memory usage(MB): 3549
�[m�[37m        
�[m�[37m        
�[m�[37m        5/15/2023 5:29:00 PM	 Time elapsed(s): 0.008
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        Suffix of length 27 compared against sequence of length 34
�[m�[37m        Running 'MulticlassLogisticRegression' on 'iris'
�[m�[37m          Running as: CV tr=MulticlassLogisticRegression{l1=0.001 l2=0.1 ot=1e-3 nt=1} data=/root/helix/work/correlation/test/data/iris.txt seed=1 xf=Term{col=Label} dout={/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-CV-iris.txt} norm=no threads-
�[m�[37m        maml.exe CV tr=MulticlassLogisticRegression{l1=0.001 l2=0.1 ot=1e-3 nt=1} threads=- norm=No dout=/root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-CV-iris.txt data=/root/helix/work/correlation/test/data/iris.txt seed=1 xf=Term{col=Label}
�[m�[37m        
�[m�[37m        
�[m�[37m        Not adding a normalizer.
�[m�[37m        
�[m�[37m        
�[m�[37m        Not adding a normalizer.
�[m�[37m        
�[m�[37m        
�[m�[37m        Beginning optimization
�[m�[37m        
�[m�[37m        
�[m�[37m        num vars: 15
�[m�[37m        
�[m�[37m        
�[m�[37m        improvement criterion: Mean Improvement
�[m�[37m        
�[m�[37m        
�[m�[37m        L1 regularization selected 15 of 15 weights.
�[m�[37m        
�[m�[37m        
�[m�[37m        Not training a calibrator because it is not needed.
�[m�[37m        
�[m�[37m        
�[m�[37m        Physical memory usage(MB): 120
�[m�[37m        
�[m�[37m        
�[m�[37m        Virtual memory usage(MB): 3549
�[m�[37m        
�[m�[37m        
�[m�[37m        5/15/2023 5:29:00 PM	 Time elapsed(s): 0.018
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        
�[m�[37m        ***** Unexpected failure. Please refer to https://aka.ms/MLNetIssue to file an issue with details *****
�[m�[37m        
�[m�[37m        
�[m�[37m        ***** Error log has been saved to '/tmp/TLC/Error_20230515_172900_e06f381a-3a62-4df3-b493-09ff5a4df87d.log', please refer to https://aka.ms/MLNetIssue to file an issue with details *****
�[m�[37m        
�[m�[37m        
�[m�[37m        ===== Begin detailed dump =====
�[m�[37m        
�[m�[37m        
�[m�[37m        (1) Unexpected exception: One or more errors occurred. (Assert failed: Assertion Failed
�[m�[37m        
�[m�[37m        
�[m�[37m        Expected: True
�[m�[37m        
�[m�[37m        
�[m�[37m        Actual:   False), 'System.AggregateException'
�[m�[37m        
�[m�[37m        
�[m�[37m           at System.Threading.Tasks.Task.WaitAllCore(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
�[m�[37m        
�[m�[37m        
�[m�[37m           at System.Threading.Tasks.Task.WaitAll(Task[] tasks)
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Data.CrossValidationCommand.FoldHelper.GetCrossValidationTasks() in /__w/1/s/src/Microsoft.ML.Data/Commands/CrossValidationCommand.cs:line 471
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Data.CrossValidationCommand.RunCore(IChannel ch, String cmd) in /__w/1/s/src/Microsoft.ML.Data/Commands/CrossValidationCommand.cs:line 202
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Data.CrossValidationCommand.Run() in /__w/1/s/src/Microsoft.ML.Data/Commands/CrossValidationCommand.cs:line 134
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Tools.Maml.MainCore(IHostEnvironment env, String args, Boolean alwaysPrintStacktrace) in /__w/1/s/src/Microsoft.ML.Maml/MAML.cs:line 130
�[m�[37m        
�[m�[37m        
�[m�[37m        (2) Unexpected exception: Assert failed: Assertion Failed
�[m�[37m        
�[m�[37m        
�[m�[37m        Expected: True
�[m�[37m        
�[m�[37m        
�[m�[37m        Actual:   False, 'Xunit.Sdk.TrueException'
�[m�[37m        
�[m�[37m        
�[m�[37m           at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
�[m�[37m        
�[m�[37m        
�[m�[37m           at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 116
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Internal.Internallearn.Test.GlobalBase.AssertHandler(String msg, IExceptionContext ectx) in /__w/1/s/test/Microsoft.ML.TestFramework/GlobalBase.cs:line 80
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Runtime.Contracts.DbgFailCore(String msg, IExceptionContext ctx) in /__w/1/s/src/Microsoft.ML.Core/Utilities/Contracts.cs:line 781
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Runtime.Contracts.DbgFail(IExceptionContext ctx) in /__w/1/s/src/Microsoft.ML.Core/Utilities/Contracts.cs:line 790
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Runtime.Contracts.Assert(IExceptionContext ctx, Boolean f) in /__w/1/s/src/Microsoft.ML.Core/Utilities/Contracts.cs:line 845
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Data.CacheDataView.ColumnCache.ImplVec`1.Fetch(Int32 idx, VBuffer`1& value) in /__w/1/s/src/Microsoft.ML.Data/DataView/CacheDataView.cs:line 1389
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Data.CacheDataView.RowCursor`1.<>c__DisplayClass8_0`1.<CreateGetterDelegateCore>b__0(TValue& value) in /__w/1/s/src/Microsoft.ML.Data/DataView/CacheDataView.cs:line 496
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Trainers.FeatureFloatVectorCursor.Accept() in /__w/1/s/src/Microsoft.ML.Data/Training/TrainerUtils.cs:line 855
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Trainers.FloatLabelCursor.Accept() in /__w/1/s/src/Microsoft.ML.Data/Training/TrainerUtils.cs:line 928
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Trainers.TrainingCursorBase.MoveNext() in /__w/1/s/src/Microsoft.ML.Data/Training/TrainerUtils.cs:line 555
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Trainers.LbfgsTrainerBase`3.TrainCore(IChannel ch, RoleMappedData data) in /__w/1/s/src/Microsoft.ML.StandardTrainers/Standard/LogisticRegression/LbfgsPredictorBase.cs:line 605
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Trainers.LbfgsTrainerBase`3.TrainModelCore(TrainContext context) in /__w/1/s/src/Microsoft.ML.StandardTrainers/Standard/LogisticRegression/LbfgsPredictorBase.cs:line 458
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Trainers.TrainerEstimatorBase`2.Microsoft.ML.ITrainer<Microsoft.ML.IPredictor>.Train(TrainContext context) in /__w/1/s/src/Microsoft.ML.Data/Training/TrainerEstimatorBase.cs:line 100
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Trainers.TrainerEstimatorBase`2.Microsoft.ML.ITrainer.Train(TrainContext context) in /__w/1/s/src/Microsoft.ML.Data/Training/TrainerEstimatorBase.cs:line 168
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Data.TrainUtils.TrainCore(IHostEnvironment env, IChannel ch, RoleMappedData data, ITrainer trainer, RoleMappedData validData, IComponentFactory`1 calibrator, Int32 maxCalibrationExamples, Nullable`1 cacheData, IPredictor inputPredictor, RoleMappedData testData) in /__w/1/s/src/Microsoft.ML.Data/Commands/TrainCommand.cs:line 280
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Data.TrainUtils.Train(IHostEnvironment env, IChannel ch, RoleMappedData data, ITrainer trainer, RoleMappedData validData, IComponentFactory`1 calibrator, Int32 maxCalibrationExamples, Nullable`1 cacheData, IPredictor inputPredictor, RoleMappedData testData) in /__w/1/s/src/Microsoft.ML.Data/Commands/TrainCommand.cs:line 255
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Data.CrossValidationCommand.FoldHelper.RunFold(Int32 fold) in /__w/1/s/src/Microsoft.ML.Data/Commands/CrossValidationCommand.cs:line 523
�[m�[37m        
�[m�[37m        
�[m�[37m           at Microsoft.ML.Data.CrossValidationCommand.FoldHelper.<>c__DisplayClass24_0.<GetCrossValidationTasks>b__0() in /__w/1/s/src/Microsoft.ML.Data/Commands/CrossValidationCommand.cs:line 463
�[m�[37m        
�[m�[37m        
�[m�[37m           at System.Threading.Tasks.Task`1.InnerInvoke()
�[m�[37m        
�[m�[37m        
�[m�[37m           at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
�[m�[37m        
�[m�[37m        
�[m�[37m        --- End of stack trace from previous location ---
�[m�[37m        
�[m�[37m        
�[m�[37m           at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
�[m�[37m        
�[m�[37m        
�[m�[37m        ====== End detailed dump =====
�[m�[37m        
�[m�[37m        
�[m�[37m        *** Predictor returned -2
�[m�[37m        Comparing /root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-CV-iris-out.txt and /root/helix/work/correlation/test/BaselineOutput/Common/MulticlassLogisticRegression/netcoreapp/MulticlassLogisticRegression-CV-iris-out.txt
�[m�[37m        *** Failure #1: Output and baseline mismatch at line 3, expected 'Beginning optimization' but got 'Not adding a normalizer.' : 'MulticlassLogisticRegression/MulticlassLogisticRegression-CV-iris-out.txt'
�[m�[37m        		Void Fail(System.String, System.Object[]) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 224
�[m�[37m        		Boolean CheckEqualityFromPathsCore(System.String, System.String, System.String, Int32, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 546
�[m�[37m        		Boolean CheckEqualityCore(System.String, System.String, System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 409
�[m�[37m        		Boolean CheckEqualityNormalized(System.String, System.String, System.String, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 377
�[m�[37m        		Boolean CheckEqualityNormalized(Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/TestCommandBase.cs 79
�[m�[37m        		Void Run(RunContext, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 169
�[m�[37m        		Void Run_CV(Microsoft.ML.RunTests.PredictorAndArgs, Microsoft.ML.TestFrameworkCommon.TestDataset, System.String[], System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 442
�[m�[37m        		Void RunOneAllTests(Microsoft.ML.RunTests.PredictorAndArgs, Microsoft.ML.TestFrameworkCommon.TestDataset, System.String[], System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 363
�[m�[37m        		Void MulticlassLRTest() /__w/1/s/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs 155
�[m�[37m        		System.Object InvokeMethod(System.Object, System.Span`1[System.Object] ByRef, System.Signature, Boolean, Boolean)  0
�[m�[37m        		System.Object Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)  0
�[m�[37m        		System.Object CallTestMethod(System.Object) /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 150
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 257
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task <InvokeTestMethodAsync>b__1()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs 48
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task AggregateAsync(System.Func`1[System.Threading.Tasks.Task])  0
�[m�[37m        		System.Threading.Tasks.Task <InvokeTestMethodAsync>b__0() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 242
�[m�[37m        		Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 90
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task RunAsync(System.Func`1[System.Threading.Tasks.Task])  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 241
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(System.Object)  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(System.Object) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestInvoker.cs 112
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 206
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Decimal] <RunAsync>b__47_0()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 107
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[T] RunAsync[T](System.Func`1[System.Threading.Tasks.Task`1[T]])  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Decimal] RunAsync() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 189
�[m�[37m        		System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(Xunit.Sdk.ExceptionAggregator) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs 84
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs 67
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Tuple`2[System.Decimal,System.String]] InvokeTestAsync(Xunit.Sdk.ExceptionAggregator)  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Tuple`2[System.Decimal,System.String]] <RunAsync>b__0() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestRunner.cs 149
�[m�[37m        		Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 107
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[T] RunAsync[T](System.Func`1[System.Threading.Tasks.Task`1[T]])  0
�[m�[37m        		Void MoveNext()  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCaseRunner.cs 82
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs 136
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestCasesAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs 106
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs 213
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestMethodsAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs 171
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestClassAsync(Xunit.Abstractions.ITestClass, Xunit.Abstractions.IReflectionTypeInfo, System.Collections.Generic.IEnumerable`1[Xunit.Sdk.IXunitTestCase]) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestCollectionRunner.cs 158
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs 130
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestClassesAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs 101
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestCollectionAsync(Xunit.Sdk.IMessageBus, Xunit.Abstractions.ITestCollection, System.Collections.Generic.IEnumerable`1[Xunit.Sdk.IXunitTestCase], System.Threading.CancellationTokenSource) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs 235
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] <RunTestCollectionsAsync>b__2() /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs 184
�[m�[37m        		Void InnerInvoke()  0
�[m�[37m        		Void <.cctor>b__272_0(System.Object)  0
�[m�[37m        		Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
�[m�[37m        		Void ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)  0
�[m�[37m        		Boolean ExecuteEntry()  0
�[m�[37m        		Void <.cctor>b__8_0(System.Object)  0
�[m�[37m        		Void RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object) /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 106
�[m�[37m        		Void <WorkerThreadProc>b__0(System.Object) /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 96
�[m�[37m        		Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
�[m�[37m        		Void WorkerThreadProc() /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 96
�[m�[37m        		Void <QueueUserWorkItem>b__5_0(System.Object) /_/src/common/XunitWorkerThread.cs 37
�[m�[37m        		Void InnerInvoke()  0
�[m�[37m        		Void <.cctor>b__272_0(System.Object)  0
�[m�[37m        		Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
�[m�[37m        		Void ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)  0
�[m�[37m        		Void ExecuteEntryUnsafe(System.Threading.Thread)  0
�[m�[37m        		Void <.cctor>b__10_0(System.Object)  0
�[m�[37m        		Void StartCallback()  0
�[m�[37m        *** Failure #2: Output file not found: /root/helix/work/workitem/e/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-CV-iris.txt
�[m�[37m        		Void Fail(System.String, System.Object[]) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 224
�[m�[37m        		Boolean CheckOutFile(System.String) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 922
�[m�[37m        		Boolean CheckEqualityCore(System.String, System.String, System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 398
�[m�[37m        		Boolean CheckEquality(System.String, System.String, System.String, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 367
�[m�[37m        		Boolean CheckEquality(Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/TestCommandBase.cs 73
�[m�[37m        		Void Run(RunContext, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 218
�[m�[37m        		Void Run_CV(Microsoft.ML.RunTests.PredictorAndArgs, Microsoft.ML.TestFrameworkCommon.TestDataset, System.String[], System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 442
�[m�[37m        		Void RunOneAllTests(Microsoft.ML.RunTests.PredictorAndArgs, Microsoft.ML.TestFrameworkCommon.TestDataset, System.String[], System.String, Boolean, Int32, NumberParseOption) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs 363
�[m�[37m        		Void MulticlassLRTest() /__w/1/s/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs 155
�[m�[37m        		System.Object InvokeMethod(System.Object, System.Span`1[System.Object] ByRef, System.Signature, Boolean, Boolean)  0
�[m�[37m        		System.Object Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)  0
�[m�[37m        		System.Object CallTestMethod(System.Object) /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 150
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 257
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task <InvokeTestMethodAsync>b__1()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs 48
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task AggregateAsync(System.Func`1[System.Threading.Tasks.Task])  0
�[m�[37m        		System.Threading.Tasks.Task <InvokeTestMethodAsync>b__0() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 242
�[m�[37m        		Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 90
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task RunAsync(System.Func`1[System.Threading.Tasks.Task])  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 241
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(System.Object)  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(System.Object) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestInvoker.cs 112
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 206
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Decimal] <RunAsync>b__47_0()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 107
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[T] RunAsync[T](System.Func`1[System.Threading.Tasks.Task`1[T]])  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Decimal] RunAsync() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs 189
�[m�[37m        		System.Threading.Tasks.Task`1[System.Decimal] InvokeTestMethodAsync(Xunit.Sdk.ExceptionAggregator) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs 84
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs 67
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Tuple`2[System.Decimal,System.String]] InvokeTestAsync(Xunit.Sdk.ExceptionAggregator)  0
�[m�[37m        		System.Threading.Tasks.Task`1[System.Tuple`2[System.Decimal,System.String]] <RunAsync>b__0() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestRunner.cs 149
�[m�[37m        		Void MoveNext() /_/src/xunit.core/Sdk/ExceptionAggregator.cs 107
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[T] RunAsync[T](System.Func`1[System.Threading.Tasks.Task`1[T]])  0
�[m�[37m        		Void MoveNext()  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCaseRunner.cs 82
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs 136
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestCasesAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs 106
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs 213
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestMethodsAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs 171
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestClassAsync(Xunit.Abstractions.ITestClass, Xunit.Abstractions.IReflectionTypeInfo, System.Collections.Generic.IEnumerable`1[Xunit.Sdk.IXunitTestCase]) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestCollectionRunner.cs 158
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs 130
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestClassesAsync()  0
�[m�[37m        		Void MoveNext() /_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs 101
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		Void Start[TStateMachine](TStateMachine ByRef)  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunAsync()  0
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] RunTestCollectionAsync(Xunit.Sdk.IMessageBus, Xunit.Abstractions.ITestCollection, System.Collections.Generic.IEnumerable`1[Xunit.Sdk.IXunitTestCase], System.Threading.CancellationTokenSource) /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs 235
�[m�[37m        		System.Threading.Tasks.Task`1[Xunit.Sdk.RunSummary] <RunTestCollectionsAsync>b__2() /_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs 184
�[m�[37m        		Void InnerInvoke()  0
�[m�[37m        		Void <.cctor>b__272_0(System.Object)  0
�[m�[37m        		Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
�[m�[37m        		Void ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)  0
�[m�[37m        		Boolean ExecuteEntry()  0
�[m�[37m        		Void <.cctor>b__8_0(System.Object)  0
�[m�[37m        		Void RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object) /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 106
�[m�[37m        		Void <WorkerThreadProc>b__0(System.Object) /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 96
�[m�[37m        		Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
�[m�[37m        		Void WorkerThreadProc() /_/src/xunit.execution/Sdk/MaxConcurrencySyncContext.cs 96
�[m�[37m        		Void <QueueUserWorkItem>b__5_0(System.Object) /_/src/common/XunitWorkerThread.cs 37
�[m�[37m        		Void InnerInvoke()  0
�[m�[37m        		Void <.cctor>b__272_0(System.Object)  0
�[m�[37m        		Void RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)  0
�[m�[37m        		Void ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)  0
�[m�[37m        		Void ExecuteEntryUnsafe(System.Threading.Thread)  0
�[m�[37m        		Void <.cctor>b__10_0(System.Object)  0
�[m�[37m        		Void StartCallback()  0
�[m�[37m        Test MulticlassLRTest: completed normally: failed
�[m�[37m        Test MulticlassLRTest is using netcoreapp configuration specific baselines.

Activity

  1. ghost added
    untriagedNew issue has not been triaged
    on May 15, 2023
  2. michaelgsharp
    added
    test
    and removed
    untriagedNew issue has not been triaged
    on May 15, 2023
  3. michaelgsharp
    added this to the ML.NET Future milestone on May 15, 2023

  4. michaelgsharp commented on May 15, 2023

    @michaelgsharp
    michaelgsharp
    on May 15, 2023
    ContributorAuthor

    Also disabling TestGridSearchTrialRunner2 for same reason.

  5. rosebyte
    added
    area-TrainersMachine learning trainers and training APIs
    and removed
    test
    on Sep 8, 2026
  6. rosebyte
    added
    greendevelopment
    on Sep 21, 2026

  7. svick commented on Nov 3, 2026

    @svick
    svick
    on Nov 3, 2026
    Member

    Triage: We should investigate the failures and restore the disabled coverage when we have time.

    The original failure includes a CacheDataView assertion during cross-validation, which might need a separate fix. Microsoft.ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2 is also involved in #6841, #6842 and #6843.

  8. svick
    added
    test-bugProblem in test source code (most likely)
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-TrainersMachine learning trainers and training APIsgreendevelopmenttest-bugProblem in test source code (most likely)

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions