Custom attribute isnt found during test collection ordering - xunit
Im running E2E tests with some of the logics written using xUnit (.NET 8, c#) and need the tests to run in particular order. Id like to control the order in 2 ways: test case and class - and it needs to be ordered by numbers (int that is). This is the test case attribute: [AttributeUsage(AttributeTargets.Method)] public class TestCaseOrderAttribute : Attribute { public int Order { get; } public TestCaseOrderAttribute(int order) { Order = order; } } and this is the class level attribute: [AttributeUsage(AttributeTargets.Class)] public class TestClassOrderAttribute : Attribute { public int Order { get; } public TestClassOrderAttribute(int order) { Order = order; } } This is the test case orderer: public class TestCaseOrderer: ITestCaseOrderer { public IEnumerable OrderTestCases(IEnumerable testCases) where TTestCase : ITestCase { return testCases.OrderBy(tc => { var orderAttribute = tc.TestMethod.Method.GetCustomAttributes(typeof(TestCaseOrderAttribute)).FirstOrDefault(); return orderAttribute?.GetNamedArgument("Order") ?? 0; }); } } and this is the class orderer: public class TestCollectionOrderer : ITestCollectionOrderer { public IEnumerable OrderTestCollections(IEnumerable testCollections) { return testCollections.OrderBy(tc => { var classOrderAttribute = tc.CollectionDefinition.GetCustomAttributes(typeof(TestClassOrderAttribute)).FirstOrDefault(); return classOrderAttribute?.GetNamedArgument("Order") ?? 0; }); } } ALL are defined in some infra project in my solution. this is the collection definition - defined in my test project (meaning the attributes and orderers are in a different project): [CollectionDefinition("TestCollection")] public class BasicTestCollectionDefinition: ICollectionFixture { } This is my assemblyInfo of the test project: [assembly: TestCollectionOrderer("Infra.TestOrder.Orderers.TestCollectionOrders.TestCollectionOrderer", "Infra")] [assembly: TestCaseOrderer("Infra.TestOrder.Orderers.TestCaseOrders.TestCaseOrderer", "Infra")] [assembly: CollectionBehavior(DisableTestParallelization = true)] and finally - these are my sample tests: [Collection("TestCollection"), TestClassOrder(2)] [Trait("test_name", "update_installation")] public class PingTest { public PingTest(ILogger logger, LinuxBasicFixture fixture) { Console.WriteLine("Running PingTest linux test"); } [Fact, TestCaseOrder(2)] public void Ping_ShouldSucceed_Second() { Assert.True(true); } [Fact, TestCaseOrder(1)] public void Ping_ShouldSucceed() { Assert.True(true); } } while the ordering of the test cases is working as expected - the class attributes arent found. meaning the line in the test collection orderer var classOrderAttribute = tc.CollectionDefinition.GetCustomAttributes(typeof(TestClassOrderAttribute)).FirstOrDefault(); returns null. The test case attr and orderer and the test collection attr and orderer are written just almost the same as you can see. No matter what i tried and what I read in here and the xUnit docs - i couldnt find what im missing. I looked at this lib (https://github.com/tomaszeman/Xunit.Extensions.Ordering/tree/master) and couldnt find anything im doing different than what its author did - and yet im missing something. I dont want to use the mentioned lib as it is not well maintained. would appreciate any help!
Im running E2E tests with some of the logics written using xUnit (.NET 8, c#) and need the tests to run in particular order. Id like to control the order in 2 ways: test case and class - and it needs to be ordered by numbers (int that is).
This is the test case attribute:
[AttributeUsage(AttributeTargets.Method)]
public class TestCaseOrderAttribute : Attribute
{
public int Order { get; }
public TestCaseOrderAttribute(int order)
{
Order = order;
}
}
and this is the class level attribute:
[AttributeUsage(AttributeTargets.Class)]
public class TestClassOrderAttribute : Attribute
{
public int Order { get; }
public TestClassOrderAttribute(int order)
{
Order = order;
}
}
This is the test case orderer:
public class TestCaseOrderer: ITestCaseOrderer
{
public IEnumerable OrderTestCases(IEnumerable testCases) where TTestCase : ITestCase
{
return testCases.OrderBy(tc =>
{
var orderAttribute = tc.TestMethod.Method.GetCustomAttributes(typeof(TestCaseOrderAttribute)).FirstOrDefault();
return orderAttribute?.GetNamedArgument("Order") ?? 0;
});
}
}
and this is the class orderer:
public class TestCollectionOrderer : ITestCollectionOrderer
{
public IEnumerable OrderTestCollections(IEnumerable testCollections)
{
return testCollections.OrderBy(tc =>
{
var classOrderAttribute = tc.CollectionDefinition.GetCustomAttributes(typeof(TestClassOrderAttribute)).FirstOrDefault();
return classOrderAttribute?.GetNamedArgument("Order") ?? 0;
});
}
}
ALL are defined in some infra
project in my solution.
this is the collection definition - defined in my test project (meaning the attributes and orderers are in a different project):
[CollectionDefinition("TestCollection")]
public class BasicTestCollectionDefinition: ICollectionFixture
{
}
This is my assemblyInfo of the test project:
[assembly: TestCollectionOrderer("Infra.TestOrder.Orderers.TestCollectionOrders.TestCollectionOrderer", "Infra")]
[assembly: TestCaseOrderer("Infra.TestOrder.Orderers.TestCaseOrders.TestCaseOrderer", "Infra")]
[assembly: CollectionBehavior(DisableTestParallelization = true)]
and finally - these are my sample tests:
[Collection("TestCollection"), TestClassOrder(2)]
[Trait("test_name", "update_installation")]
public class PingTest
{
public PingTest(ILogger logger, LinuxBasicFixture fixture)
{
Console.WriteLine("Running PingTest linux test");
}
[Fact, TestCaseOrder(2)]
public void Ping_ShouldSucceed_Second()
{
Assert.True(true);
}
[Fact, TestCaseOrder(1)]
public void Ping_ShouldSucceed()
{
Assert.True(true);
}
}
while the ordering of the test cases is working as expected - the class attributes arent found. meaning the line in the test collection orderer var classOrderAttribute = tc.CollectionDefinition.GetCustomAttributes(typeof(TestClassOrderAttribute)).FirstOrDefault();
returns null.
The test case attr and orderer and the test collection attr and orderer are written just almost the same as you can see. No matter what i tried and what I read in here and the xUnit docs - i couldnt find what im missing. I looked at this lib (https://github.com/tomaszeman/Xunit.Extensions.Ordering/tree/master) and couldnt find anything im doing different than what its author did - and yet im missing something. I dont want to use the mentioned lib as it is not well maintained.
would appreciate any help!