diff --git a/docs/plan/Unity式Library资产导入与缓存系统重构方案.md b/docs/plan/Unity式Library资产导入与缓存系统重构方案.md index 6473d7af..f9182898 100644 --- a/docs/plan/Unity式Library资产导入与缓存系统重构方案.md +++ b/docs/plan/Unity式Library资产导入与缓存系统重构方案.md @@ -14,8 +14,8 @@ | 阶段 3:TextureImporter | 已完成 | 已支持纹理导入为 `xctex`,运行时可直接从 artifact 读取,不再总是解码原始图片。 | | 阶段 4:ModelImporter | 已完成(初版) | 已支持模型导入为 `xcmesh`,并缓存模型关联纹理的 `xctex`;`ResourceManager` 会优先走 artifact。 | | 阶段 5:Material 系统 lazy 引用化 | 部分完成 | 已补齐材质序列化所需的 tag/property/texture binding 访问接口,但仍是 eager 贴图加载,未完成真正的延迟纹理解析。 | -| 阶段 6:Scene / Component 引用迁移 | 部分完成 | `MeshFilterComponent` 与 `MeshRendererComponent` 已开始双写路径和 `AssetRef`,可兼容旧格式读取。Scene 全量格式迁移尚未完成。 | -| 阶段 7:异步导入与异步加载 | 未开始 | 当前导入与大资源加载仍发生在主线程,是仍会卡 editor 的直接原因。 | +| 阶段 6:Scene / Component 引用迁移 | 部分完成 | `MeshFilterComponent` 与 `MeshRendererComponent` 已开始双写路径和 `AssetRef`,并已在 editor 场景打开路径接入延迟恢复。Scene 全量格式迁移尚未完成。 | +| 阶段 7:异步导入与异步加载 | 部分完成 | `AsyncLoader` 已具备真实 worker 线程与主线程 completion pump;editor 打开 scene 时已改为 deferred restore,但通用导入队列、placeholder/UI、全资源类型异步化尚未完成。 | | 阶段 8:清理、GC、工具与可视化 | 未开始 | 还没有 Reimport All、依赖图查看、orphan artifact 清理、导入面板等工具。 | ### 0.2 本轮已经落地的具体内容 @@ -25,18 +25,21 @@ - `TextureLoader` 已支持 `xctex`,`MeshLoader` 已支持 `xcmesh`,从而把“导入”和“运行时加载”从代码路径上拆开。 - `ProjectManager` 已支持忽略 `.meta` 文件显示,并在资源删除、移动、重命名时携带 `.meta` sidecar。 - `MeshFilterComponent` 与 `MeshRendererComponent` 已开始从纯路径引用迁到 `AssetRef`,目前是“旧路径兼容 + 新引用双写”模式。 +- `AsyncLoader` 已从空壳改为真实后台加载器,支持 worker thread、完成队列与主线程回调泵。 +- editor 的 `SceneManager::LoadScene(...)` / `RestoreSceneSnapshot(...)` 已接入 deferred scene load scope,scene 打开时不再在组件反序列化阶段同步恢复 mesh/material。 +- `EditorWorkspace::Update(...)` 已开始逐帧 pump async completion,`MeshFilterComponent` / `MeshRendererComponent` 现在会在资源真正完成后再接管 handle。 - 已为纹理 artifact、模型 artifact、组件 `AssetRef` 序列化补充回归测试。 ### 0.3 已验证结果 - `Debug` 下 `texture_tests.exe` 已通过,覆盖 `.meta` 自动生成、`Library` 产物生成、二次 `EnsureArtifact` 不重复导入、按 `AssetRef` 再加载纹理。 -- `Debug` 下 `components_tests.exe` 已通过,覆盖 `MeshRendererComponent` 的 `AssetRef` 双写序列化/反序列化,以及项目内材质按 `AssetRef` 恢复。 +- `Debug` 下 `components_tests.exe` 已通过,覆盖 `MeshRendererComponent` 的 `AssetRef` 双写序列化/反序列化,以及项目内材质按 `AssetRef` 的 deferred async 恢复。 - `Release` 下 `mesh_tests.exe` 已通过,覆盖 `obj + mtl + 贴图` 导入为 `xcmesh + xctex`、二次导入命中缓存、按 `AssetRef` 再加载模型。 - `Release` 下 `XCEditor` 已构建通过。 ### 0.4 当前版本仍然存在的限制 -- 首次打开未缓存或失效的 `obj` 资源时,仍会在主线程同步执行导入,editor 会明显卡住。 +- editor 场景打开阶段虽然已经不再在组件反序列化里同步恢复大资源,但首次真正需要该资源时,若 artifact 缺失或失效,后台任务仍可能较重,且缺少显式进度/UI。 - 即使已经命中 artifact,当前模型 artifact 读取后仍会立即把关联贴图一起加载到内存,尚未做到材质纹理 lazy load。 - 首帧渲染阶段仍会发生 GPU 资源创建与上传,这部分也会带来一次性卡顿。 - `MaterialImporter` 还没有形成完整的独立 artifact 格式,材质系统仍处于过渡状态。 @@ -1447,8 +1450,11 @@ editor/src/AssetPipeline/ 当前状态: -- 未开始。 -- 这是当前 editor 仍会被大 `obj` 场景卡住的首要剩余任务。 +- 部分完成。 +- `AsyncLoader` 已具备真实 worker thread、pending/completed queue、主线程 completion callback pump。 +- editor `SceneManager` 已在 `LoadScene(...)` 与 `RestoreSceneSnapshot(...)` 路径启用 deferred scene load scope。 +- `MeshFilterComponent` 与 `MeshRendererComponent` 已支持“反序列化只恢复引用并提交异步请求,真正 handle 接管延后到完成回调之后”。 +- 仍未完成通用 import job 系统、统一 placeholder 状态、批量队列调度、更多资源类型的异步恢复,以及失败/进度 UI。 ## 阶段 8:清理、GC、工具与可视化 diff --git a/engine/include/XCEngine/Core/Asset/ArtifactFormats.h b/engine/include/XCEngine/Core/Asset/ArtifactFormats.h index 32b5ca82..8baab5cb 100644 --- a/engine/include/XCEngine/Core/Asset/ArtifactFormats.h +++ b/engine/include/XCEngine/Core/Asset/ArtifactFormats.h @@ -10,7 +10,8 @@ namespace XCEngine { namespace Resources { constexpr Core::uint32 kTextureArtifactSchemaVersion = 1; -constexpr Core::uint32 kMeshArtifactSchemaVersion = 1; +constexpr Core::uint32 kMaterialArtifactSchemaVersion = 1; +constexpr Core::uint32 kMeshArtifactSchemaVersion = 2; struct TextureArtifactHeader { char magic[8] = { 'X', 'C', 'T', 'E', 'X', '0', '1', '\0' }; @@ -26,7 +27,7 @@ struct TextureArtifactHeader { }; struct MeshArtifactHeader { - char magic[8] = { 'X', 'C', 'M', 'E', 'S', 'H', '1', '\0' }; + char magic[8] = { 'X', 'C', 'M', 'E', 'S', 'H', '2', '\0' }; Core::uint32 schemaVersion = kMeshArtifactSchemaVersion; Core::uint32 vertexCount = 0; Core::uint32 vertexStride = 0; @@ -35,13 +36,17 @@ struct MeshArtifactHeader { Core::uint32 use32BitIndex = 0; Core::uint32 sectionCount = 0; Core::uint32 materialCount = 0; - Core::uint32 textureCount = 0; Math::Vector3 boundsMin = Math::Vector3::Zero(); Math::Vector3 boundsMax = Math::Vector3::Zero(); Core::uint64 vertexDataSize = 0; Core::uint64 indexDataSize = 0; }; +struct MaterialArtifactFileHeader { + char magic[8] = { 'X', 'C', 'M', 'A', 'T', '0', '1', '\0' }; + Core::uint32 schemaVersion = kMaterialArtifactSchemaVersion; +}; + struct MaterialArtifactHeader { Core::int32 renderQueue = static_cast(MaterialRenderQueue::Geometry); MaterialRenderState renderState = {}; diff --git a/engine/include/XCEngine/Core/Asset/AssetDatabase.h b/engine/include/XCEngine/Core/Asset/AssetDatabase.h index e4628f92..1bb6db62 100644 --- a/engine/include/XCEngine/Core/Asset/AssetDatabase.h +++ b/engine/include/XCEngine/Core/Asset/AssetDatabase.h @@ -72,7 +72,7 @@ public: const Containers::String& GetLibraryRoot() const { return m_libraryRoot; } private: - static constexpr Core::uint32 kCurrentImporterVersion = 1; + static constexpr Core::uint32 kCurrentImporterVersion = 2; void EnsureProjectLayout(); void LoadSourceAssetDB(); @@ -105,6 +105,8 @@ private: ArtifactRecord& outRecord); bool ImportTextureAsset(const SourceAssetRecord& sourceRecord, ArtifactRecord& outRecord); + bool ImportMaterialAsset(const SourceAssetRecord& sourceRecord, + ArtifactRecord& outRecord); bool ImportModelAsset(const SourceAssetRecord& sourceRecord, ArtifactRecord& outRecord); diff --git a/engine/include/XCEngine/Resources/Material/Material.h b/engine/include/XCEngine/Resources/Material/Material.h index 526cb4d0..222e7a71 100644 --- a/engine/include/XCEngine/Resources/Material/Material.h +++ b/engine/include/XCEngine/Resources/Material/Material.h @@ -10,6 +10,7 @@ #include #include #include +#include #include namespace XCEngine { @@ -132,10 +133,18 @@ struct MaterialTagEntry { Containers::String value; }; +struct PendingTextureLoadState { + IResource* resource = nullptr; + Containers::String errorMessage; + bool completed = false; +}; + struct MaterialTextureBinding { Containers::String name; Core::uint32 slot = 0; ResourceHandle texture; + Containers::String texturePath; + std::shared_ptr pendingLoad; }; class Material : public IResource { @@ -181,6 +190,7 @@ public: void SetInt(const Containers::String& name, Core::int32 value); void SetBool(const Containers::String& name, bool value); void SetTexture(const Containers::String& name, const ResourceHandle& texture); + void SetTexturePath(const Containers::String& name, const Containers::String& texturePath); float GetFloat(const Containers::String& name) const; Math::Vector2 GetFloat2(const Containers::String& name) const; @@ -191,6 +201,7 @@ public: ResourceHandle GetTexture(const Containers::String& name) const; Core::uint32 GetTextureBindingCount() const { return static_cast(m_textureBindings.Size()); } Containers::String GetTextureBindingName(Core::uint32 index) const; + Containers::String GetTextureBindingPath(Core::uint32 index) const; ResourceHandle GetTextureBindingTexture(Core::uint32 index) const; const Containers::Array& GetTextureBindings() const { return m_textureBindings; } std::vector GetProperties() const; @@ -205,6 +216,9 @@ public: void ClearAllProperties(); private: + void BeginAsyncTextureLoad(Core::uint32 index); + void ResolvePendingTextureBinding(Core::uint32 index); + void ResolvePendingTextureBindings(); void MarkChanged(bool updateConstantBuffer); void UpdateMemorySize(); diff --git a/engine/src/Core/Asset/AssetDatabase.cpp b/engine/src/Core/Asset/AssetDatabase.cpp index 7f495c57..7bcd7a1c 100644 --- a/engine/src/Core/Asset/AssetDatabase.cpp +++ b/engine/src/Core/Asset/AssetDatabase.cpp @@ -2,7 +2,9 @@ #include #include +#include #include +#include #include #include @@ -39,6 +41,13 @@ Containers::String ToContainersString(const std::string& value) { return Containers::String(value.c_str()); } +Containers::String NormalizeArtifactPathString(const Containers::String& path) { + if (path.Empty()) { + return Containers::String(); + } + return ToContainersString(fs::path(path.CStr()).lexically_normal().generic_string()); +} + std::string TrimCopy(const std::string& text) { const auto begin = std::find_if_not(text.begin(), text.end(), [](unsigned char ch) { return std::isspace(ch) != 0; @@ -200,11 +209,45 @@ std::vector GatherMaterialProperties(const Material& material) return material.GetProperties(); } -void WriteMaterialBlock(std::ofstream& output, - const Material& material, - const std::unordered_map& textureFileNames) { +Containers::String ResolveTextureBindingPath( + const Material& material, + Core::uint32 bindingIndex, + const std::unordered_map& textureArtifactPaths) { + const ResourceHandle textureHandle = material.GetTextureBindingTexture(bindingIndex); + const Texture* texture = textureHandle.Get(); + if (texture != nullptr) { + const auto textureIt = textureArtifactPaths.find(texture); + if (textureIt != textureArtifactPaths.end()) { + return textureIt->second; + } + + if (!texture->GetPath().Empty()) { + return NormalizeArtifactPathString(texture->GetPath()); + } + } + + return NormalizeArtifactPathString(material.GetTextureBindingPath(bindingIndex)); +} + +bool WriteMaterialArtifactFile( + const fs::path& artifactPath, + const Material& material, + const std::unordered_map& textureArtifactPaths = {}) { + std::ofstream output(artifactPath, std::ios::binary | std::ios::trunc); + if (!output.is_open()) { + return false; + } + + MaterialArtifactFileHeader fileHeader; + output.write(reinterpret_cast(&fileHeader), sizeof(fileHeader)); + WriteString(output, material.GetName()); WriteString(output, material.GetPath()); + WriteString( + output, + material.GetShader() != nullptr + ? material.GetShader()->GetPath() + : Containers::String()); WriteString(output, material.GetShaderPass()); MaterialArtifactHeader header; @@ -243,18 +286,17 @@ void WriteMaterialBlock(std::ofstream& output, for (Core::uint32 bindingIndex = 0; bindingIndex < material.GetTextureBindingCount(); ++bindingIndex) { const Containers::String bindingName = material.GetTextureBindingName(bindingIndex); - const Texture* texture = material.GetTextureBindingTexture(bindingIndex).Get(); - auto fileIt = texture != nullptr ? textureFileNames.find(texture) : textureFileNames.end(); WriteString(output, bindingName); - WriteString(output, - fileIt != textureFileNames.end() - ? ToContainersString(fileIt->second) - : Containers::String()); + WriteString(output, ResolveTextureBindingPath(material, bindingIndex, textureArtifactPaths)); } + + return static_cast(output); } -bool WriteMeshArtifactFile(const fs::path& artifactPath, const Mesh& mesh) { +bool WriteMeshArtifactFile(const fs::path& artifactPath, + const Mesh& mesh, + const std::vector& materialArtifactPaths) { std::ofstream output(artifactPath, std::ios::binary | std::ios::trunc); if (!output.is_open()) { return false; @@ -267,8 +309,7 @@ bool WriteMeshArtifactFile(const fs::path& artifactPath, const Mesh& mesh) { header.indexCount = mesh.GetIndexCount(); header.use32BitIndex = mesh.IsUse32BitIndex() ? 1u : 0u; header.sectionCount = static_cast(mesh.GetSections().Size()); - header.materialCount = static_cast(mesh.GetMaterials().Size()); - header.textureCount = static_cast(mesh.GetTextures().Size()); + header.materialCount = static_cast(materialArtifactPaths.size()); header.boundsMin = mesh.GetBounds().GetMin(); header.boundsMax = mesh.GetBounds().GetMax(); header.vertexDataSize = static_cast(mesh.GetVertexDataSize()); @@ -286,25 +327,8 @@ bool WriteMeshArtifactFile(const fs::path& artifactPath, const Mesh& mesh) { output.write(static_cast(mesh.GetIndexData()), mesh.GetIndexDataSize()); } - std::unordered_map textureFileNames; - for (size_t textureIndex = 0; textureIndex < mesh.GetTextures().Size(); ++textureIndex) { - const Texture* texture = mesh.GetTextures()[textureIndex]; - if (texture == nullptr) { - continue; - } - - const std::string fileName = "texture_" + std::to_string(textureIndex) + ".xctex"; - textureFileNames.emplace(texture, fileName); - WriteString(output, ToContainersString(fileName)); - } - - for (size_t materialIndex = 0; materialIndex < mesh.GetMaterials().Size(); ++materialIndex) { - const Material* material = mesh.GetMaterials()[materialIndex]; - const Core::uint32 materialPresent = material != nullptr ? 1u : 0u; - output.write(reinterpret_cast(&materialPresent), sizeof(materialPresent)); - if (material != nullptr) { - WriteMaterialBlock(output, *material, textureFileNames); - } + for (const Containers::String& materialArtifactPath : materialArtifactPaths) { + WriteString(output, NormalizeArtifactPathString(materialArtifactPath)); } return static_cast(output); @@ -852,7 +876,8 @@ bool AssetDatabase::ShouldReimport(const SourceAssetRecord& sourceRecord, return true; } - return artifactRecord->sourceHash != sourceRecord.sourceHash || + return artifactRecord->importerVersion != sourceRecord.importerVersion || + artifactRecord->sourceHash != sourceRecord.sourceHash || artifactRecord->metaHash != sourceRecord.metaHash || artifactRecord->sourceFileSize != sourceRecord.sourceFileSize || artifactRecord->sourceWriteTime != sourceRecord.sourceWriteTime; @@ -864,6 +889,8 @@ bool AssetDatabase::ImportAsset(const SourceAssetRecord& sourceRecord, switch (primaryType) { case ResourceType::Texture: return ImportTextureAsset(sourceRecord, outRecord); + case ResourceType::Material: + return ImportMaterialAsset(sourceRecord, outRecord); case ResourceType::Mesh: return ImportModelAsset(sourceRecord, outRecord); default: @@ -1034,6 +1061,53 @@ bool AssetDatabase::ImportTextureAsset(const SourceAssetRecord& sourceRecord, return true; } +bool AssetDatabase::ImportMaterialAsset(const SourceAssetRecord& sourceRecord, + ArtifactRecord& outRecord) { + MaterialLoader loader; + const Containers::String absolutePath = + NormalizePathString(fs::path(m_projectRoot.CStr()) / sourceRecord.relativePath.CStr()); + LoadResult result = loader.Load(absolutePath); + if (!result || result.resource == nullptr) { + return false; + } + + Material* material = static_cast(result.resource); + const Containers::String artifactKey = BuildArtifactKey(sourceRecord); + const Containers::String artifactDir = BuildArtifactDirectory(artifactKey); + const Containers::String mainArtifactPath = + NormalizePathString(fs::path(artifactDir.CStr()) / "main.xcmat"); + + std::error_code ec; + fs::remove_all(fs::path(m_projectRoot.CStr()) / artifactDir.CStr(), ec); + ec.clear(); + fs::create_directories(fs::path(m_projectRoot.CStr()) / artifactDir.CStr(), ec); + if (ec) { + delete material; + return false; + } + + const bool writeOk = + WriteMaterialArtifactFile(fs::path(m_projectRoot.CStr()) / mainArtifactPath.CStr(), *material); + delete material; + if (!writeOk) { + return false; + } + + outRecord.artifactKey = artifactKey; + outRecord.assetGuid = sourceRecord.guid; + outRecord.importerName = sourceRecord.importerName; + outRecord.importerVersion = sourceRecord.importerVersion; + outRecord.resourceType = ResourceType::Material; + outRecord.artifactDirectory = artifactDir; + outRecord.mainArtifactPath = mainArtifactPath; + outRecord.sourceHash = sourceRecord.sourceHash; + outRecord.metaHash = sourceRecord.metaHash; + outRecord.sourceFileSize = sourceRecord.sourceFileSize; + outRecord.sourceWriteTime = sourceRecord.sourceWriteTime; + outRecord.mainLocalID = kMainAssetLocalID; + return true; +} + bool AssetDatabase::ImportModelAsset(const SourceAssetRecord& sourceRecord, ArtifactRecord& outRecord) { MeshLoader loader; @@ -1057,18 +1131,54 @@ bool AssetDatabase::ImportModelAsset(const SourceAssetRecord& sourceRecord, return false; } - bool writeOk = WriteMeshArtifactFile(fs::path(m_projectRoot.CStr()) / mainArtifactPath.CStr(), *mesh); + bool writeOk = true; + std::unordered_map textureArtifactPaths; for (size_t textureIndex = 0; writeOk && textureIndex < mesh->GetTextures().Size(); ++textureIndex) { Texture* texture = mesh->GetTextures()[textureIndex]; if (texture == nullptr) { continue; } - const fs::path textureArtifactPath = - fs::path(m_projectRoot.CStr()) / artifactDir.CStr() / ("texture_" + std::to_string(textureIndex) + ".xctex"); - writeOk = WriteTextureArtifactFile(textureArtifactPath, *texture); + const Containers::String textureArtifactPath = + NormalizePathString(fs::path(artifactDir.CStr()) / ("texture_" + std::to_string(textureIndex) + ".xctex")); + writeOk = WriteTextureArtifactFile( + fs::path(m_projectRoot.CStr()) / textureArtifactPath.CStr(), + *texture); + if (!writeOk) { + break; + } + + textureArtifactPaths.emplace(texture, textureArtifactPath); } + std::vector materialArtifactPaths; + materialArtifactPaths.reserve(mesh->GetMaterials().Size()); + for (size_t materialIndex = 0; writeOk && materialIndex < mesh->GetMaterials().Size(); ++materialIndex) { + Material* material = mesh->GetMaterials()[materialIndex]; + if (material == nullptr) { + materialArtifactPaths.emplace_back(); + continue; + } + + const Containers::String materialArtifactPath = + NormalizePathString(fs::path(artifactDir.CStr()) / ("material_" + std::to_string(materialIndex) + ".xcmat")); + writeOk = WriteMaterialArtifactFile( + fs::path(m_projectRoot.CStr()) / materialArtifactPath.CStr(), + *material, + textureArtifactPaths); + if (!writeOk) { + break; + } + + materialArtifactPaths.push_back(materialArtifactPath); + } + + writeOk = writeOk && + WriteMeshArtifactFile( + fs::path(m_projectRoot.CStr()) / mainArtifactPath.CStr(), + *mesh, + materialArtifactPaths); + DestroyImportedMesh(mesh); if (!writeOk) { return false; diff --git a/engine/src/Resources/Material/Material.cpp b/engine/src/Resources/Material/Material.cpp index 8eb82d56..476db7f1 100644 --- a/engine/src/Resources/Material/Material.cpp +++ b/engine/src/Resources/Material/Material.cpp @@ -52,6 +52,15 @@ void RemoveTextureBindingByName( } } +void EnsureTextureProperty(Containers::HashMap& properties, + const Containers::String& name) { + MaterialProperty prop; + prop.name = name; + prop.type = MaterialPropertyType::Texture; + prop.refCount = 1; + properties.Insert(name, prop); +} + void WritePackedMaterialProperty(Core::uint8* destination, const MaterialProperty& property) { std::memset(destination, 0, kMaterialConstantSlotSize); @@ -271,15 +280,13 @@ void Material::SetBool(const Containers::String& name, bool value) { } void Material::SetTexture(const Containers::String& name, const ResourceHandle& texture) { - MaterialProperty prop; - prop.name = name; - prop.type = MaterialPropertyType::Texture; - prop.refCount = 1; - m_properties.Insert(name, prop); + EnsureTextureProperty(m_properties, name); for (auto& binding : m_textureBindings) { if (binding.name == name) { binding.texture = texture; + binding.texturePath = texture.Get() != nullptr ? texture->GetPath() : Containers::String(); + binding.pendingLoad.reset(); MarkChanged(false); return; } @@ -289,6 +296,33 @@ void Material::SetTexture(const Containers::String& name, const ResourceHandle(m_textureBindings.Size()); binding.texture = texture; + binding.texturePath = texture.Get() != nullptr ? texture->GetPath() : Containers::String(); + m_textureBindings.PushBack(binding); + MarkChanged(false); +} + +void Material::SetTexturePath(const Containers::String& name, const Containers::String& texturePath) { + if (texturePath.Empty()) { + RemoveProperty(name); + return; + } + + EnsureTextureProperty(m_properties, name); + + for (auto& binding : m_textureBindings) { + if (binding.name == name) { + binding.texture.Reset(); + binding.texturePath = texturePath; + binding.pendingLoad.reset(); + MarkChanged(false); + return; + } + } + + MaterialTextureBinding binding; + binding.name = name; + binding.slot = static_cast(m_textureBindings.Size()); + binding.texturePath = texturePath; m_textureBindings.PushBack(binding); MarkChanged(false); } @@ -343,8 +377,16 @@ bool Material::GetBool(const Containers::String& name) const { } ResourceHandle Material::GetTexture(const Containers::String& name) const { - for (const auto& binding : m_textureBindings) { + Material* material = const_cast(this); + material->ResolvePendingTextureBindings(); + for (Core::uint32 bindingIndex = 0; bindingIndex < material->m_textureBindings.Size(); ++bindingIndex) { + MaterialTextureBinding& binding = material->m_textureBindings[bindingIndex]; if (binding.name == name) { + if (binding.texture.Get() == nullptr && + binding.pendingLoad == nullptr && + !binding.texturePath.Empty()) { + material->BeginAsyncTextureLoad(bindingIndex); + } return binding.texture; } } @@ -355,8 +397,24 @@ Containers::String Material::GetTextureBindingName(Core::uint32 index) const { return index < m_textureBindings.Size() ? m_textureBindings[index].name : Containers::String(); } +Containers::String Material::GetTextureBindingPath(Core::uint32 index) const { + return index < m_textureBindings.Size() ? m_textureBindings[index].texturePath : Containers::String(); +} + ResourceHandle Material::GetTextureBindingTexture(Core::uint32 index) const { - return index < m_textureBindings.Size() ? m_textureBindings[index].texture : ResourceHandle(); + Material* material = const_cast(this); + material->ResolvePendingTextureBinding(index); + if (index < material->m_textureBindings.Size()) { + MaterialTextureBinding& binding = material->m_textureBindings[index]; + if (binding.texture.Get() == nullptr && + binding.pendingLoad == nullptr && + !binding.texturePath.Empty()) { + material->BeginAsyncTextureLoad(index); + } + return binding.texture; + } + + return ResourceHandle(); } std::vector Material::GetProperties() const { @@ -404,6 +462,59 @@ void Material::RecalculateMemorySize() { UpdateMemorySize(); } +void Material::BeginAsyncTextureLoad(Core::uint32 index) { + if (index >= m_textureBindings.Size()) { + return; + } + + MaterialTextureBinding& binding = m_textureBindings[index]; + if (binding.texture.Get() != nullptr || binding.texturePath.Empty() || binding.pendingLoad != nullptr) { + return; + } + + binding.pendingLoad = std::make_shared(); + std::weak_ptr weakState = binding.pendingLoad; + const Containers::String texturePath = binding.texturePath; + ResourceManager::Get().LoadAsync( + texturePath, + ResourceType::Texture, + [weakState](LoadResult result) { + if (std::shared_ptr state = weakState.lock()) { + state->resource = result.resource; + state->errorMessage = result.errorMessage; + state->completed = true; + } + }); +} + +void Material::ResolvePendingTextureBinding(Core::uint32 index) { + if (index >= m_textureBindings.Size()) { + return; + } + + MaterialTextureBinding& binding = m_textureBindings[index]; + if (!binding.pendingLoad || !binding.pendingLoad->completed) { + return; + } + + std::shared_ptr completedLoad = std::move(binding.pendingLoad); + binding.pendingLoad.reset(); + if (completedLoad->resource == nullptr) { + return; + } + + binding.texture = ResourceHandle(static_cast(completedLoad->resource)); + if (binding.texture.Get() != nullptr) { + binding.texturePath = binding.texture->GetPath(); + } +} + +void Material::ResolvePendingTextureBindings() { + for (Core::uint32 bindingIndex = 0; bindingIndex < m_textureBindings.Size(); ++bindingIndex) { + ResolvePendingTextureBinding(bindingIndex); + } +} + bool Material::HasProperty(const Containers::String& name) const { return m_properties.Contains(name); } @@ -457,6 +568,7 @@ void Material::UpdateMemorySize() { for (const auto& binding : m_textureBindings) { m_memorySize += binding.name.Length(); + m_memorySize += binding.texturePath.Length(); } } diff --git a/engine/src/Resources/Material/MaterialLoader.cpp b/engine/src/Resources/Material/MaterialLoader.cpp index 44dd16c2..a6f88964 100644 --- a/engine/src/Resources/Material/MaterialLoader.cpp +++ b/engine/src/Resources/Material/MaterialLoader.cpp @@ -1,12 +1,15 @@ #include +#include #include #include #include #include #include +#include #include #include +#include #include namespace XCEngine { @@ -18,6 +21,107 @@ std::string ToStdString(const Containers::Array& data) { return std::string(reinterpret_cast(data.Data()), data.Size()); } +Containers::Array ReadMaterialArtifactFileData(const Containers::String& path) { + Containers::Array data; + + auto tryRead = [&data](const std::filesystem::path& filePath, bool& opened) { + std::ifstream file(filePath, std::ios::binary | std::ios::ate); + if (!file.is_open()) { + opened = false; + return; + } + + opened = true; + const std::streamsize size = file.tellg(); + if (size <= 0) { + return; + } + + file.seekg(0, std::ios::beg); + data.Resize(static_cast(size)); + if (!file.read(reinterpret_cast(data.Data()), size)) { + data.Clear(); + } + }; + + bool opened = false; + const std::filesystem::path inputPath(path.CStr()); + tryRead(inputPath, opened); + if (opened || path.Empty() || inputPath.is_absolute()) { + return data; + } + + const Containers::String& resourceRoot = ResourceManager::Get().GetResourceRoot(); + if (resourceRoot.Empty()) { + return data; + } + + tryRead(std::filesystem::path(resourceRoot.CStr()) / inputPath, opened); + return data; +} + +Containers::String NormalizePathString(const std::filesystem::path& path) { + return Containers::String(path.lexically_normal().generic_string().c_str()); +} + +Containers::String ResolveArtifactDependencyPath(const Containers::String& dependencyPath, + const Containers::String& ownerArtifactPath) { + if (dependencyPath.Empty()) { + return dependencyPath; + } + + std::filesystem::path dependencyFsPath(dependencyPath.CStr()); + if (dependencyFsPath.is_absolute() && std::filesystem::exists(dependencyFsPath)) { + return NormalizePathString(dependencyFsPath); + } + + if (std::filesystem::exists(dependencyFsPath)) { + return NormalizePathString(dependencyFsPath); + } + + const Containers::String& resourceRoot = ResourceManager::Get().GetResourceRoot(); + if (!resourceRoot.Empty()) { + const std::filesystem::path projectRelativeCandidate = + std::filesystem::path(resourceRoot.CStr()) / dependencyFsPath; + if (std::filesystem::exists(projectRelativeCandidate)) { + return NormalizePathString(projectRelativeCandidate); + } + } + + const std::filesystem::path ownerArtifactFsPath(ownerArtifactPath.CStr()); + if (!ownerArtifactFsPath.is_absolute()) { + return dependencyPath; + } + + const std::filesystem::path ownerRelativeCandidate = + ownerArtifactFsPath.parent_path() / dependencyFsPath; + if (std::filesystem::exists(ownerRelativeCandidate)) { + return NormalizePathString(ownerRelativeCandidate); + } + + std::filesystem::path current = ownerArtifactFsPath.parent_path(); + while (!current.empty()) { + if (current.filename() == "Library") { + const std::filesystem::path projectRoot = current.parent_path(); + if (!projectRoot.empty()) { + const std::filesystem::path projectRelativeCandidate = projectRoot / dependencyFsPath; + if (std::filesystem::exists(projectRelativeCandidate)) { + return NormalizePathString(projectRelativeCandidate); + } + } + break; + } + + const std::filesystem::path parent = current.parent_path(); + if (parent == current) { + break; + } + current = parent; + } + + return dependencyPath; +} + size_t SkipWhitespace(const std::string& text, size_t pos) { while (pos < text.size() && std::isspace(static_cast(text[pos])) != 0) { ++pos; @@ -551,6 +655,181 @@ bool MaterialFileExists(const Containers::String& path) { return std::filesystem::exists(std::filesystem::path(resourceRoot.CStr()) / inputPath); } +ResourceHandle LoadShaderHandle(const Containers::String& shaderPath); + +template +bool ReadMaterialArtifactValue(const Containers::Array& data, size_t& offset, T& outValue) { + if (offset + sizeof(T) > data.Size()) { + return false; + } + + std::memcpy(&outValue, data.Data() + offset, sizeof(T)); + offset += sizeof(T); + return true; +} + +bool ReadMaterialArtifactString(const Containers::Array& data, + size_t& offset, + Containers::String& outValue) { + Core::uint32 length = 0; + if (!ReadMaterialArtifactValue(data, offset, length)) { + return false; + } + + if (length == 0) { + outValue.Clear(); + return true; + } + + if (offset + length > data.Size()) { + return false; + } + + outValue = Containers::String( + std::string(reinterpret_cast(data.Data() + offset), length).c_str()); + offset += length; + return true; +} + +void ApplyMaterialProperty(Material& material, const MaterialProperty& property) { + switch (property.type) { + case MaterialPropertyType::Float: + material.SetFloat(property.name, property.value.floatValue[0]); + break; + case MaterialPropertyType::Float2: + material.SetFloat2( + property.name, + Math::Vector2(property.value.floatValue[0], property.value.floatValue[1])); + break; + case MaterialPropertyType::Float3: + material.SetFloat3( + property.name, + Math::Vector3( + property.value.floatValue[0], + property.value.floatValue[1], + property.value.floatValue[2])); + break; + case MaterialPropertyType::Float4: + material.SetFloat4( + property.name, + Math::Vector4( + property.value.floatValue[0], + property.value.floatValue[1], + property.value.floatValue[2], + property.value.floatValue[3])); + break; + case MaterialPropertyType::Int: + material.SetInt(property.name, property.value.intValue[0]); + break; + case MaterialPropertyType::Bool: + material.SetBool(property.name, property.value.boolValue); + break; + default: + break; + } +} + +LoadResult LoadMaterialArtifact(const Containers::String& path) { + const Containers::Array data = ReadMaterialArtifactFileData(path); + if (data.Empty()) { + return LoadResult("Failed to read material artifact: " + path); + } + + size_t offset = 0; + MaterialArtifactFileHeader fileHeader; + if (!ReadMaterialArtifactValue(data, offset, fileHeader)) { + return LoadResult("Failed to parse material artifact header: " + path); + } + + const std::string magic(fileHeader.magic, fileHeader.magic + 7); + if (magic != "XCMAT01") { + return LoadResult("Invalid material artifact magic: " + path); + } + + auto material = std::make_unique(); + material->m_path = path; + material->m_name = path; + material->m_guid = ResourceGUID::Generate(path); + + Containers::String materialName; + Containers::String materialSourcePath; + Containers::String shaderPath; + Containers::String shaderPass; + if (!ReadMaterialArtifactString(data, offset, materialName) || + !ReadMaterialArtifactString(data, offset, materialSourcePath) || + !ReadMaterialArtifactString(data, offset, shaderPath) || + !ReadMaterialArtifactString(data, offset, shaderPass)) { + return LoadResult("Failed to parse material artifact strings: " + path); + } + + material->m_name = materialName.Empty() ? path : materialName; + if (!materialSourcePath.Empty()) { + material->m_path = materialSourcePath; + material->m_guid = ResourceGUID::Generate(materialSourcePath); + } + + if (!shaderPath.Empty()) { + const ResourceHandle shaderHandle = LoadShaderHandle(shaderPath); + if (shaderHandle.IsValid()) { + material->SetShader(shaderHandle); + } + } + if (!shaderPass.Empty()) { + material->SetShaderPass(shaderPass); + } + + MaterialArtifactHeader header; + if (!ReadMaterialArtifactValue(data, offset, header)) { + return LoadResult("Failed to parse material artifact body: " + path); + } + + material->SetRenderQueue(header.renderQueue); + material->SetRenderState(header.renderState); + + for (Core::uint32 tagIndex = 0; tagIndex < header.tagCount; ++tagIndex) { + Containers::String tagName; + Containers::String tagValue; + if (!ReadMaterialArtifactString(data, offset, tagName) || + !ReadMaterialArtifactString(data, offset, tagValue)) { + return LoadResult("Failed to read material artifact tags: " + path); + } + + material->SetTag(tagName, tagValue); + } + + for (Core::uint32 propertyIndex = 0; propertyIndex < header.propertyCount; ++propertyIndex) { + Containers::String propertyName; + MaterialPropertyArtifact propertyArtifact; + if (!ReadMaterialArtifactString(data, offset, propertyName) || + !ReadMaterialArtifactValue(data, offset, propertyArtifact)) { + return LoadResult("Failed to read material artifact properties: " + path); + } + + MaterialProperty property; + property.name = propertyName; + property.type = static_cast(propertyArtifact.propertyType); + property.value = propertyArtifact.value; + ApplyMaterialProperty(*material, property); + } + + for (Core::uint32 bindingIndex = 0; bindingIndex < header.textureBindingCount; ++bindingIndex) { + Containers::String bindingName; + Containers::String texturePath; + if (!ReadMaterialArtifactString(data, offset, bindingName) || + !ReadMaterialArtifactString(data, offset, texturePath)) { + return LoadResult("Failed to read material artifact texture bindings: " + path); + } + + if (!texturePath.Empty()) { + material->SetTexturePath(bindingName, ResolveArtifactDependencyPath(texturePath, path)); + } + } + + material->m_isValid = true; + material->RecalculateMemorySize(); + return LoadResult(material.release()); +} + } // namespace MaterialLoader::MaterialLoader() = default; @@ -562,6 +841,7 @@ Containers::Array MaterialLoader::GetSupportedExtensions() c extensions.PushBack("mat"); extensions.PushBack("material"); extensions.PushBack("json"); + extensions.PushBack("xcmat"); return extensions; } @@ -570,8 +850,8 @@ bool MaterialLoader::CanLoad(const Containers::String& path) const { return true; } - Containers::String ext = GetExtension(path); - return ext == "mat" || ext == "material" || ext == "json"; + Containers::String ext = GetExtension(path).ToLower(); + return ext == "mat" || ext == "material" || ext == "json" || ext == "xcmat"; } LoadResult MaterialLoader::Load(const Containers::String& path, const ImportSettings* settings) { @@ -581,6 +861,11 @@ LoadResult MaterialLoader::Load(const Containers::String& path, const ImportSett return CreateBuiltinMaterialResource(path); } + const Containers::String ext = GetExtension(path).ToLower(); + if (ext == "xcmat") { + return LoadMaterialArtifact(path); + } + Containers::Array data = ReadFileData(path); Material* material = new Material(); material->m_path = path; diff --git a/engine/src/Resources/Mesh/MeshLoader.cpp b/engine/src/Resources/Mesh/MeshLoader.cpp index 7ee132fd..67c59210 100644 --- a/engine/src/Resources/Mesh/MeshLoader.cpp +++ b/engine/src/Resources/Mesh/MeshLoader.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include #include @@ -148,6 +149,68 @@ Core::uint32 FindEmbeddedTextureIndex(const aiScene& scene, const aiTexture& emb return 0; } +Containers::String NormalizePathString(const std::filesystem::path& path) { + return Containers::String(path.lexically_normal().generic_string().c_str()); +} + +Containers::String ResolveArtifactDependencyPath(const Containers::String& dependencyPath, + const Containers::String& ownerArtifactPath) { + if (dependencyPath.Empty()) { + return dependencyPath; + } + + std::filesystem::path dependencyFsPath(dependencyPath.CStr()); + if (dependencyFsPath.is_absolute() && std::filesystem::exists(dependencyFsPath)) { + return NormalizePathString(dependencyFsPath); + } + + if (std::filesystem::exists(dependencyFsPath)) { + return NormalizePathString(dependencyFsPath); + } + + const Containers::String& resourceRoot = ResourceManager::Get().GetResourceRoot(); + if (!resourceRoot.Empty()) { + const std::filesystem::path projectRelativeCandidate = + std::filesystem::path(resourceRoot.CStr()) / dependencyFsPath; + if (std::filesystem::exists(projectRelativeCandidate)) { + return NormalizePathString(projectRelativeCandidate); + } + } + + const std::filesystem::path ownerArtifactFsPath(ownerArtifactPath.CStr()); + if (!ownerArtifactFsPath.is_absolute()) { + return dependencyPath; + } + + const std::filesystem::path ownerRelativeCandidate = + ownerArtifactFsPath.parent_path() / dependencyFsPath; + if (std::filesystem::exists(ownerRelativeCandidate)) { + return NormalizePathString(ownerRelativeCandidate); + } + + std::filesystem::path current = ownerArtifactFsPath.parent_path(); + while (!current.empty()) { + if (current.filename() == "Library") { + const std::filesystem::path projectRoot = current.parent_path(); + if (!projectRoot.empty()) { + const std::filesystem::path projectRelativeCandidate = projectRoot / dependencyFsPath; + if (std::filesystem::exists(projectRelativeCandidate)) { + return NormalizePathString(projectRelativeCandidate); + } + } + break; + } + + const std::filesystem::path parent = current.parent_path(); + if (parent == current) { + break; + } + current = parent; + } + + return dependencyPath; +} + Texture* CreateRawTexture(const Containers::String& texturePath, TextureFormat format, Core::uint32 width, @@ -570,7 +633,7 @@ LoadResult LoadMeshArtifact(const Containers::String& path) { } const std::string magic(header.magic, header.magic + 7); - if (magic != "XCMESH1") { + if (magic != "XCMESH2") { return LoadResult(Containers::String("Invalid mesh artifact magic: ") + path); } @@ -628,93 +691,31 @@ LoadResult LoadMeshArtifact(const Containers::String& path) { bounds.SetMinMax(header.boundsMin, header.boundsMax); mesh->SetBounds(bounds); - std::vector textureFiles; - textureFiles.reserve(header.textureCount); - for (Core::uint32 textureIndex = 0; textureIndex < header.textureCount; ++textureIndex) { - textureFiles.push_back(ReadBinaryString(input)); - } - - std::unordered_map loadedTextures; - TextureLoader textureLoader; - const std::filesystem::path artifactDirectory = std::filesystem::path(path.CStr()).parent_path(); + MaterialLoader materialLoader; for (Core::uint32 materialIndex = 0; materialIndex < header.materialCount; ++materialIndex) { - Core::uint32 materialPresent = 0; - input.read(reinterpret_cast(&materialPresent), sizeof(materialPresent)); + const Containers::String materialArtifactPath = ReadBinaryString(input); if (!input) { - return LoadResult(Containers::String("Failed to read mesh material flag: ") + path); + return LoadResult(Containers::String("Failed to read mesh material artifact path: ") + path); } - if (materialPresent == 0) { + if (materialArtifactPath.Empty()) { mesh->AddMaterial(nullptr); continue; } - auto* material = new Material(); - material->m_name = ReadBinaryString(input); - material->m_path = ReadBinaryString(input); - material->m_guid = ResourceGUID::Generate(material->m_path); - material->m_isValid = true; - material->SetShaderPass(ReadBinaryString(input)); - - MaterialArtifactHeader materialHeader; - input.read(reinterpret_cast(&materialHeader), sizeof(materialHeader)); - if (!input) { - delete material; - return LoadResult(Containers::String("Failed to read material artifact header: ") + path); - } - - material->SetRenderQueue(materialHeader.renderQueue); - material->SetRenderState(materialHeader.renderState); - - for (Core::uint32 tagIndex = 0; tagIndex < materialHeader.tagCount; ++tagIndex) { - material->SetTag(ReadBinaryString(input), ReadBinaryString(input)); - } - - for (Core::uint32 propertyIndex = 0; propertyIndex < materialHeader.propertyCount; ++propertyIndex) { - MaterialProperty property; - property.name = ReadBinaryString(input); - - MaterialPropertyArtifact propertyArtifact; - input.read(reinterpret_cast(&propertyArtifact), sizeof(propertyArtifact)); - if (!input) { - delete material; - return LoadResult(Containers::String("Failed to read material property: ") + path); - } - - property.type = static_cast(propertyArtifact.propertyType); - property.value = propertyArtifact.value; - ApplyMaterialProperty(*material, property); - } - - for (Core::uint32 bindingIndex = 0; bindingIndex < materialHeader.textureBindingCount; ++bindingIndex) { - const Containers::String bindingName = ReadBinaryString(input); - const Containers::String textureFile = ReadBinaryString(input); - if (textureFile.Empty()) { - continue; - } - - const std::string textureKey(textureFile.CStr()); - Texture* texture = nullptr; - auto textureIt = loadedTextures.find(textureKey); - if (textureIt != loadedTextures.end()) { - texture = textureIt->second; - } else { - const Containers::String texturePath = - Containers::String((artifactDirectory / textureFile.CStr()).lexically_normal().string().c_str()); - LoadResult textureResult = textureLoader.Load(texturePath); - if (textureResult && textureResult.resource != nullptr) { - texture = static_cast(textureResult.resource); - loadedTextures.emplace(textureKey, texture); - mesh->AddTexture(texture); - } - } - - if (texture != nullptr) { - material->SetTexture(bindingName, ResourceHandle(texture)); - } + const Containers::String resolvedMaterialArtifactPath = + ResolveArtifactDependencyPath(materialArtifactPath, path); + LoadResult materialResult = materialLoader.Load(resolvedMaterialArtifactPath); + if (!materialResult || materialResult.resource == nullptr) { + return LoadResult( + Containers::String("Failed to load mesh material artifact: ") + + resolvedMaterialArtifactPath + + " for " + + path); } + auto* material = static_cast(materialResult.resource); material->RecalculateMemorySize(); mesh->AddMaterial(material); } diff --git a/engine/src/Resources/Texture/TextureLoader.cpp b/engine/src/Resources/Texture/TextureLoader.cpp index 510042cb..548495fe 100644 --- a/engine/src/Resources/Texture/TextureLoader.cpp +++ b/engine/src/Resources/Texture/TextureLoader.cpp @@ -51,7 +51,15 @@ LoadResult CreateTextureResource(const Containers::String& path, } LoadResult LoadTextureArtifact(const Containers::String& path) { - std::ifstream input(path.CStr(), std::ios::binary); + std::filesystem::path resolvedPath(path.CStr()); + if (!resolvedPath.is_absolute() && !std::filesystem::exists(resolvedPath)) { + const Containers::String& resourceRoot = ResourceManager::Get().GetResourceRoot(); + if (!resourceRoot.Empty()) { + resolvedPath = std::filesystem::path(resourceRoot.CStr()) / resolvedPath; + } + } + + std::ifstream input(resolvedPath, std::ios::binary); if (!input.is_open()) { return LoadResult(Containers::String("Failed to read texture artifact: ") + path); } diff --git a/tests/Resources/Material/CMakeLists.txt b/tests/Resources/Material/CMakeLists.txt index af14dc23..56091c0f 100644 --- a/tests/Resources/Material/CMakeLists.txt +++ b/tests/Resources/Material/CMakeLists.txt @@ -27,5 +27,9 @@ target_include_directories(material_tests PRIVATE ${CMAKE_SOURCE_DIR}/tests/Fixtures ) +target_compile_definitions(material_tests PRIVATE + XCENGINE_TEST_FIXTURES_DIR="${CMAKE_SOURCE_DIR}/tests/Fixtures" +) + include(GoogleTest) gtest_discover_tests(material_tests) diff --git a/tests/Resources/Material/test_material_loader.cpp b/tests/Resources/Material/test_material_loader.cpp index 2e7c5083..a4a0c006 100644 --- a/tests/Resources/Material/test_material_loader.cpp +++ b/tests/Resources/Material/test_material_loader.cpp @@ -1,18 +1,46 @@ #include +#include +#include #include #include #include #include +#include #include #include #include +#include using namespace XCEngine::Resources; using namespace XCEngine::Containers; namespace { +std::string GetMeshFixturePath(const char* fileName) { + return (std::filesystem::path(XCENGINE_TEST_FIXTURES_DIR) / "Resources" / "Mesh" / fileName).string(); +} + +void WriteArtifactString(std::ofstream& output, const XCEngine::Containers::String& value) { + const XCEngine::Core::uint32 length = static_cast(value.Length()); + output.write(reinterpret_cast(&length), sizeof(length)); + if (length > 0) { + output.write(value.CStr(), length); + } +} + +bool PumpAsyncLoadsUntilIdle(ResourceManager& manager, + std::chrono::milliseconds timeout = std::chrono::milliseconds(4000)) { + const auto deadline = std::chrono::steady_clock::now() + timeout; + while (manager.IsAsyncLoading() && std::chrono::steady_clock::now() < deadline) { + manager.UpdateAsyncLoads(); + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + + manager.UpdateAsyncLoads(); + return !manager.IsAsyncLoading(); +} + TEST(MaterialLoader, GetResourceType) { MaterialLoader loader; EXPECT_EQ(loader.GetResourceType(), ResourceType::Material); @@ -28,6 +56,7 @@ TEST(MaterialLoader, CanLoad) { MaterialLoader loader; EXPECT_TRUE(loader.CanLoad("test.mat")); EXPECT_TRUE(loader.CanLoad("test.json")); + EXPECT_TRUE(loader.CanLoad("test.xcmat")); EXPECT_FALSE(loader.CanLoad("test.txt")); EXPECT_FALSE(loader.CanLoad("test.png")); } @@ -184,4 +213,107 @@ TEST(MaterialLoader, ResourceManagerLoadsRelativeMaterialFromResourceRoot) { manager.Shutdown(); } +TEST(MaterialLoader, AssetDatabaseCreatesMaterialArtifact) { + namespace fs = std::filesystem; + + const fs::path projectRoot = fs::temp_directory_path() / "xc_material_artifact_test"; + const fs::path assetsDir = projectRoot / "Assets"; + const fs::path materialPath = assetsDir / "relative.material"; + + fs::remove_all(projectRoot); + fs::create_directories(assetsDir); + + { + std::ofstream materialFile(materialPath); + ASSERT_TRUE(materialFile.is_open()); + materialFile << "{\n"; + materialFile << " \"renderQueue\": \"geometry\",\n"; + materialFile << " \"renderState\": {\n"; + materialFile << " \"cull\": \"back\"\n"; + materialFile << " }\n"; + materialFile << "}"; + } + + AssetDatabase database; + database.Initialize(projectRoot.string().c_str()); + + AssetDatabase::ResolvedAsset resolvedAsset; + ASSERT_TRUE(database.EnsureArtifact("Assets/relative.material", ResourceType::Material, resolvedAsset)); + EXPECT_TRUE(resolvedAsset.artifactReady); + EXPECT_EQ(fs::path(resolvedAsset.artifactMainPath.CStr()).extension().string(), ".xcmat"); + EXPECT_TRUE(fs::exists(resolvedAsset.artifactMainPath.CStr())); + + database.Shutdown(); + fs::remove_all(projectRoot); +} + +TEST(MaterialLoader, LoadMaterialArtifactDefersTexturePayloadUntilRequested) { + namespace fs = std::filesystem; + + ResourceManager& manager = ResourceManager::Get(); + manager.Initialize(); + + const fs::path projectRoot = fs::temp_directory_path() / "xc_material_xcmat_lazy_texture_test"; + const fs::path assetsDir = projectRoot / "Assets"; + const fs::path libraryDir = projectRoot / "Library" / "Manual"; + const fs::path materialArtifactPath = libraryDir / "test.xcmat"; + + fs::remove_all(projectRoot); + fs::create_directories(assetsDir); + fs::create_directories(libraryDir); + fs::copy_file( + GetMeshFixturePath("checker.bmp"), + assetsDir / "checker.bmp", + fs::copy_options::overwrite_existing); + + { + std::ofstream output(materialArtifactPath, std::ios::binary | std::ios::trunc); + ASSERT_TRUE(output.is_open()); + + MaterialArtifactFileHeader fileHeader; + output.write(reinterpret_cast(&fileHeader), sizeof(fileHeader)); + WriteArtifactString(output, "LazyMaterial"); + WriteArtifactString(output, "Assets/lazy.material"); + WriteArtifactString(output, ""); + WriteArtifactString(output, ""); + + MaterialArtifactHeader header; + header.renderQueue = static_cast(MaterialRenderQueue::Geometry); + header.textureBindingCount = 1; + output.write(reinterpret_cast(&header), sizeof(header)); + + WriteArtifactString(output, "baseColorTexture"); + WriteArtifactString(output, "Assets/checker.bmp"); + } + + manager.SetResourceRoot(projectRoot.string().c_str()); + + MaterialLoader loader; + LoadResult result = loader.Load("Library/Manual/test.xcmat"); + ASSERT_TRUE(result); + ASSERT_NE(result.resource, nullptr); + + auto* material = static_cast(result.resource); + ASSERT_NE(material, nullptr); + EXPECT_EQ(material->GetTextureBindingCount(), 1u); + EXPECT_EQ( + fs::path(material->GetTextureBindingPath(0).CStr()).lexically_normal().generic_string(), + (projectRoot / "Assets" / "checker.bmp").lexically_normal().generic_string()); + + const ResourceHandle initialTexture = material->GetTexture("baseColorTexture"); + EXPECT_FALSE(initialTexture.IsValid()); + EXPECT_GT(manager.GetAsyncPendingCount(), 0u); + + ASSERT_TRUE(PumpAsyncLoadsUntilIdle(manager)); + const ResourceHandle loadedTexture = material->GetTexture("baseColorTexture"); + ASSERT_TRUE(loadedTexture.IsValid()); + EXPECT_EQ(loadedTexture->GetWidth(), 2u); + EXPECT_EQ(loadedTexture->GetHeight(), 2u); + + delete material; + manager.SetResourceRoot(""); + manager.Shutdown(); + fs::remove_all(projectRoot); +} + } // namespace diff --git a/tests/Resources/Mesh/test_mesh_loader.cpp b/tests/Resources/Mesh/test_mesh_loader.cpp index 730aa3a4..0adad05f 100644 --- a/tests/Resources/Mesh/test_mesh_loader.cpp +++ b/tests/Resources/Mesh/test_mesh_loader.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include #include @@ -21,6 +22,35 @@ std::string GetMeshFixturePath(const char* fileName) { return (std::filesystem::path(XCENGINE_TEST_FIXTURES_DIR) / "Resources" / "Mesh" / fileName).string(); } +XCEngine::Core::uint32 GetFirstSectionMaterialIndex(const Mesh& mesh) { + if (mesh.GetSections().Empty()) { + return 0; + } + + return mesh.GetSections()[0].materialID; +} + +Material* GetFirstSectionMaterial(Mesh& mesh) { + const XCEngine::Core::uint32 materialIndex = GetFirstSectionMaterialIndex(mesh); + if (materialIndex >= mesh.GetMaterials().Size()) { + return nullptr; + } + + return mesh.GetMaterial(materialIndex); +} + +bool PumpAsyncLoadsUntilIdle(ResourceManager& manager, + std::chrono::milliseconds timeout = std::chrono::milliseconds(4000)) { + const auto deadline = std::chrono::steady_clock::now() + timeout; + while (manager.IsAsyncLoading() && std::chrono::steady_clock::now() < deadline) { + manager.UpdateAsyncLoads(); + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + + manager.UpdateAsyncLoads(); + return !manager.IsAsyncLoading(); +} + TEST(MeshLoader, GetResourceType) { MeshLoader loader; EXPECT_EQ(loader.GetResourceType(), ResourceType::Mesh); @@ -151,6 +181,9 @@ TEST(MeshLoader, AssetDatabaseCreatesModelArtifactAndReusesItWithoutReimport) { namespace fs = std::filesystem; using namespace std::chrono_literals; + ResourceManager& manager = ResourceManager::Get(); + manager.Initialize(); + const fs::path projectRoot = fs::temp_directory_path() / "xc_mesh_library_cache_test"; const fs::path assetsDir = projectRoot / "Assets"; @@ -166,6 +199,22 @@ TEST(MeshLoader, AssetDatabaseCreatesModelArtifactAndReusesItWithoutReimport) { assetsDir / "checker.bmp", fs::copy_options::overwrite_existing); + MeshLoader sourceMeshLoader; + LoadResult sourceMeshResult = + sourceMeshLoader.Load((assetsDir / "textured_triangle.obj").string().c_str()); + ASSERT_TRUE(sourceMeshResult); + ASSERT_NE(sourceMeshResult.resource, nullptr); + auto* sourceMesh = static_cast(sourceMeshResult.resource); + ASSERT_NE(sourceMesh, nullptr); + ASSERT_GE(sourceMesh->GetMaterials().Size(), 1u); + Material* sourceSectionMaterial = GetFirstSectionMaterial(*sourceMesh); + ASSERT_NE(sourceSectionMaterial, nullptr); + const XCEngine::Core::uint32 sourceMaterialIndex = GetFirstSectionMaterialIndex(*sourceMesh); + EXPECT_EQ(sourceSectionMaterial->GetTextureBindingCount(), 1u); + EXPECT_EQ(sourceSectionMaterial->GetTextureBindingName(0), "baseColorTexture"); + EXPECT_FALSE(sourceSectionMaterial->GetTextureBindingPath(0).Empty()); + delete sourceMesh; + AssetDatabase database; database.Initialize(projectRoot.string().c_str()); @@ -177,8 +226,54 @@ TEST(MeshLoader, AssetDatabaseCreatesModelArtifactAndReusesItWithoutReimport) { EXPECT_TRUE(fs::exists(projectRoot / "Library" / "SourceAssetDB" / "assets.db")); EXPECT_TRUE(fs::exists(projectRoot / "Library" / "ArtifactDB" / "artifacts.db")); EXPECT_TRUE(fs::exists(firstResolve.artifactMainPath.CStr())); + EXPECT_TRUE(fs::exists((fs::path(firstResolve.artifactDirectory.CStr()) / + ("material_" + std::to_string(sourceMaterialIndex) + ".xcmat")))); EXPECT_TRUE(fs::exists((fs::path(firstResolve.artifactDirectory.CStr()) / "texture_0.xctex"))); + MaterialLoader materialLoader; + LoadResult materialArtifactResult = + materialLoader.Load((fs::path(firstResolve.artifactDirectory.CStr()) / + ("material_" + std::to_string(sourceMaterialIndex) + ".xcmat")).string().c_str()); + ASSERT_TRUE(materialArtifactResult); + ASSERT_NE(materialArtifactResult.resource, nullptr); + auto* artifactMaterial = static_cast(materialArtifactResult.resource); + ASSERT_NE(artifactMaterial, nullptr); + EXPECT_EQ(artifactMaterial->GetTextureBindingCount(), 1u); + EXPECT_EQ(artifactMaterial->GetTextureBindingName(0), "baseColorTexture"); + EXPECT_FALSE(artifactMaterial->GetTextureBindingPath(0).Empty()); + const ResourceHandle artifactLazyTexture = artifactMaterial->GetTexture("baseColorTexture"); + EXPECT_FALSE(artifactLazyTexture.IsValid()); + EXPECT_GT(manager.GetAsyncPendingCount(), 0u); + ASSERT_TRUE(PumpAsyncLoadsUntilIdle(manager)); + const ResourceHandle artifactResolvedTexture = artifactMaterial->GetTexture("baseColorTexture"); + ASSERT_TRUE(artifactResolvedTexture.IsValid()); + EXPECT_EQ(artifactResolvedTexture->GetWidth(), 2u); + EXPECT_EQ(artifactResolvedTexture->GetHeight(), 2u); + delete artifactMaterial; + + MeshLoader meshLoader; + LoadResult meshArtifactResult = meshLoader.Load(firstResolve.artifactMainPath.CStr()); + ASSERT_TRUE(meshArtifactResult); + ASSERT_NE(meshArtifactResult.resource, nullptr); + auto* artifactMesh = static_cast(meshArtifactResult.resource); + ASSERT_NE(artifactMesh, nullptr); + ASSERT_GE(artifactMesh->GetMaterials().Size(), 1u); + Material* artifactSectionMaterial = GetFirstSectionMaterial(*artifactMesh); + ASSERT_NE(artifactSectionMaterial, nullptr); + EXPECT_EQ(artifactSectionMaterial->GetTextureBindingCount(), 1u); + EXPECT_EQ(artifactSectionMaterial->GetTextureBindingName(0), "baseColorTexture"); + EXPECT_FALSE(artifactSectionMaterial->GetTextureBindingPath(0).Empty()); + const ResourceHandle artifactMeshLazyTexture = artifactSectionMaterial->GetTexture("baseColorTexture"); + EXPECT_FALSE(artifactMeshLazyTexture.IsValid()); + EXPECT_GT(manager.GetAsyncPendingCount(), 0u); + ASSERT_TRUE(PumpAsyncLoadsUntilIdle(manager)); + const ResourceHandle artifactMeshResolvedTexture = + artifactSectionMaterial->GetTexture("baseColorTexture"); + ASSERT_TRUE(artifactMeshResolvedTexture.IsValid()); + EXPECT_EQ(artifactMeshResolvedTexture->GetWidth(), 2u); + EXPECT_EQ(artifactMeshResolvedTexture->GetHeight(), 2u); + delete artifactMesh; + AssetRef assetRef; ASSERT_TRUE(database.TryGetAssetRef("Assets/textured_triangle.obj", ResourceType::Mesh, assetRef)); EXPECT_TRUE(assetRef.IsValid()); @@ -192,6 +287,7 @@ TEST(MeshLoader, AssetDatabaseCreatesModelArtifactAndReusesItWithoutReimport) { EXPECT_EQ(originalArtifactWriteTime, fs::last_write_time(secondResolve.artifactMainPath.CStr())); database.Shutdown(); + manager.Shutdown(); fs::remove_all(projectRoot); } @@ -223,8 +319,23 @@ TEST(MeshLoader, ResourceManagerLoadsModelByAssetRefFromProjectAssets) { EXPECT_EQ(firstHandle->GetVertexCount(), 3u); EXPECT_EQ(firstHandle->GetIndexCount(), 3u); EXPECT_GE(firstHandle->GetMaterials().Size(), 1u); - EXPECT_EQ(firstHandle->GetTextures().Size(), 1u); + EXPECT_EQ(firstHandle->GetTextures().Size(), 0u); const auto initialMaterialCount = firstHandle->GetMaterials().Size(); + const XCEngine::Core::uint32 firstSectionMaterialIndex = GetFirstSectionMaterialIndex(*firstHandle.Get()); + EXPECT_LT(firstSectionMaterialIndex, initialMaterialCount); + Material* firstMaterial = GetFirstSectionMaterial(*firstHandle.Get()); + ASSERT_NE(firstMaterial, nullptr); + EXPECT_EQ(firstMaterial->GetTextureBindingCount(), 1u); + EXPECT_EQ(firstMaterial->GetTextureBindingName(0), "baseColorTexture"); + EXPECT_FALSE(firstMaterial->GetTextureBindingPath(0).Empty()); + const ResourceHandle firstLazyTexture = firstMaterial->GetTexture("baseColorTexture"); + EXPECT_FALSE(firstLazyTexture.IsValid()); + EXPECT_GT(manager.GetAsyncPendingCount(), 0u); + ASSERT_TRUE(PumpAsyncLoadsUntilIdle(manager)); + const ResourceHandle firstResolvedTexture = firstMaterial->GetTexture("baseColorTexture"); + ASSERT_TRUE(firstResolvedTexture.IsValid()); + EXPECT_EQ(firstResolvedTexture->GetWidth(), 2u); + EXPECT_EQ(firstResolvedTexture->GetHeight(), 2u); AssetRef assetRef; ASSERT_TRUE(manager.TryGetAssetRef("Assets/textured_triangle.obj", ResourceType::Mesh, assetRef)); @@ -238,7 +349,21 @@ TEST(MeshLoader, ResourceManagerLoadsModelByAssetRefFromProjectAssets) { EXPECT_EQ(secondHandle->GetVertexCount(), 3u); EXPECT_EQ(secondHandle->GetIndexCount(), 3u); EXPECT_EQ(secondHandle->GetMaterials().Size(), initialMaterialCount); - EXPECT_EQ(secondHandle->GetTextures().Size(), 1u); + EXPECT_EQ(secondHandle->GetTextures().Size(), 0u); + EXPECT_EQ(GetFirstSectionMaterialIndex(*secondHandle.Get()), firstSectionMaterialIndex); + Material* secondMaterial = GetFirstSectionMaterial(*secondHandle.Get()); + ASSERT_NE(secondMaterial, nullptr); + EXPECT_EQ(secondMaterial->GetTextureBindingCount(), 1u); + EXPECT_EQ(secondMaterial->GetTextureBindingName(0), "baseColorTexture"); + EXPECT_FALSE(secondMaterial->GetTextureBindingPath(0).Empty()); + const ResourceHandle secondLazyTexture = secondMaterial->GetTexture("baseColorTexture"); + EXPECT_FALSE(secondLazyTexture.IsValid()); + EXPECT_GT(manager.GetAsyncPendingCount(), 0u); + ASSERT_TRUE(PumpAsyncLoadsUntilIdle(manager)); + const ResourceHandle secondResolvedTexture = secondMaterial->GetTexture("baseColorTexture"); + ASSERT_TRUE(secondResolvedTexture.IsValid()); + EXPECT_EQ(secondResolvedTexture->GetWidth(), 2u); + EXPECT_EQ(secondResolvedTexture->GetHeight(), 2u); manager.SetResourceRoot(""); manager.Shutdown();